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To  the  Reader 

This  report  describes  the  activities  involved  in  conducting  SEI-assisted  assessments  of  an 
organization’s  software  process  and  provides  some  background  information  about  the  as¬ 
sessment  process  as  well.  Each  assessment  step  is  described  in  enough  detail  to  enable 
the  reader  to  understand  each  activity  and  the  purpose  behind  it;  however,  this  report  is  not 
a  substitute  for  the  necessary  training  required  to  perform  an  assessment. 

The  report  is  addressed  to  managers  and  practitioners  in  organizations  ihat  are  interested  in 
having  an  SEI-assisted  assessment  of  their  software  process.  It  is  also  appropriate  reading 
for  anyone  in  an  organization  that  is  scheduled  for  an  SEI-assisted  assessment,  particularly 
those  who  will  be  team  members  or  participants. 

The  introduction  provides  background  information  about  software  process  assessment  and 
the  tools  and  methodology  used  by  the  SEI,  along  with  an  overview  of  the  assessment  proc¬ 
ess.  Chapter  2  discusses  the  principles  underlying  assessments.  Those  who  may  be  con¬ 
sidering  having  an  assessment  will  find  these  two  sections  most  helpful,  aKig  with  the  glos¬ 
sary  and  question-and-answer  section  that  follow  the  final  chapter  of  the  report. 

Chapters  3-8  contain  a  chronological  description  of  the  assessment  process,  from  becoming 
a  candidate  for  assessment  to  implementing  an  action  plan  based  on  the  results  of  the  as¬ 
sessment.  This  detailed  view  of  assessment  will  be  of  most  interest  to  those  who  will  ac¬ 
tually  be  involved  in  an  SEI-assisted  assessment,  especially  the  team  members  and  site 
assessment  coordinator.  Please  note  that  although  assessment  is  part  of  a  larger  effort 
toward  software  process  improvement,  a  discussion  of  that  larger  effort  is  beyond  the  scope 
of  this  report. 

Many  of  the  terms  in  this  report  are  defined  in  the  ANSI/IEEE  Standard  Glossary  of  Software 
Engineering  Terminology,  729-1983.  For  those  that  are  not,  entries  appear  in  the  glossary 
at  the  end  of  this  report.  For  the  reader's  convenience,  key  or  frequently  used  ANSI/IEEE 
terms  also  appear  in  the  glossary. 

As  the  SEI  gains  knowledge  and  experience  with  the  assessment  methodology,  this  report 
will  be  updated  and  released.  If  you  have  any  comments  or  suggestions  regarding  this  re¬ 
port,  please  write  or  call  one  of  the  authors  at  this  address: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
5000  Forbes  Avenue 
Pittsburgh,  PA  15213 
(412)  268-7700 

Attn:  Software  Process  Assessment  Project 


Conducting  SEI-Assisted 
Software  Process  Assessments 


Abstract:  This  report  describes  software  process  assessment  as  it  is  performed 
in  organizations  with  the  assistance  of  the  Software  Engineering  institute  (SEI).  A 
software  process  assessment  is  an  appraisal  or  review  of  an  organization’s  soft¬ 
ware  process  (e.g.,  software  development  process).  The  main  objectives  of  such 
an  assessment  are  to  understand  the  state  of  practice  in  an  organization,  to  iden¬ 
tify  key  areas  for  improvement,  and  to  initiate  the  actions  that  facilitate  those  im¬ 
provements.  This  report  is  specifically  addressed  to  the  organizations  and  as¬ 
sessment  team  members  that  may  be  involved  in  an  SEI-assisted  software  proc¬ 
ess  assessment.  J7 .  i.  -  v  -  .  •, 

'  I 


1.  Introduction 

1.1.  Software  Process  Assessment 

Process  is  defined,  in  the  most  general  sense,  as  a  set  of  actions,  tasks,  and  procedures 
that  when  performed  or  executed  obtain  a  specific  goal  or  objective.  More  specifically,  a 
software  process  is  a  process  to  develop,  maintain,  support,  or  enhance  software.  An  ex¬ 
ample  of  a  software  process  is  a  software  development  process.  A  software  process  as¬ 
sessment  is  an  appraisal  or  review  of  an  organization’s  software  process. 

The  SEI  has  developed  and  is  currently  refining  a  methodology  for  assessing  software  proc¬ 
ess.  The  main  objectives  of  assessments  are  to  understand  the  state  of  practice  of  the 
software  process  in  an  organization,  to  identify  key  areas  for  improvement,  and  to  mitiate 
actions  that  facilitate  those  improvements.  The  purpose  of  this  report  is  to  give  a  descriptive 
overview  of  SEI-assisted  software  process  assessments.  SEI-assisted  assessments  are 
those  in  which  the  SEI  provides  consulting  and  is  directly  involved  with  an  organization  in 
performing  the  assessment. 

The  SEI  assessment  methodology  uses  a  software  process  maturity  framework  and  a  ma¬ 
turity  questionnaire.  The  next  three  sections  give  a  brief  overview  of  the  software  process 
maturity  framework,  the  maturity  questionnaire,  and  contexts  for  using  the  questionnaire. 
The  reports  referenced  in  these  sections  (and  the  assessment  training)  provide  additional, 
more  detailed  information. 
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1 .2.  Software  Process  Maturity  Framework 

A  software  process  maturity  framework  was  developed  by  the  SEI  for  two  purposes:  to  pro¬ 
vide  the  Department  of  Defense  (DoD)  with  a  means  of  characterizing  the  software  process, 
and  to  help  determine  and  improve  the  capabilities  of  software  development  organizations 
(see  reference  [18]).  The  maturity  framework  provides  the  basis  for  the  assessment.  It 
helps  identify  the  state  of  an  organization’s  software  process,  helps  provide  a  vision  of  the 
desired  process,  and  serves  as  a  mechanism  for  establishing  priorities  for  software  process 
improvement.  The  framework  is  intended  for  use  with  a  software  process  assessment 
methodology,  which  includes  the  maturity  questionnaire  described  in  the  next  section. 

The  SEI  has  defined  five  levels  of  software  process  maturity: 

1 .  Initial  —  at  the  initial  process  level,  an  organization  can  be  characterized  as 
having  ad  hoc,  or  possibly  chaotic,  process.  Typical  problems  at  this  level  are 
cost  and  schedule  overruns,  lack  of  formal  procedures,  and  little  or  no  change 
control. 

2.  Repeatable  —  the  organization  has  achieved  a  stable  process  with  a  repeat- 
able  level  of  control  by  initiating  rigorous  project  management  of  commit¬ 
ments,  costs,  schedule,  and  changes.  The  repeatable  level  has  achieved 
rudimentary  predictability  of  schedules  and  costs,  project  management,  basic 
software  configuration  management  (change  control),  and  basic  software 
quality  assurance.  A  major  problem  at  the  repeatable  level  is  that  the  software 
process  is  not  yet  completely  defined  or  recorded. 

3.  Defined  —  the  organization  has  defined  the  software  process  as  a  basis  for 
consistent  implementation  and  better  understanding.  At  this  point,  advanced 
technology  can  be  introduced  to  improve  the  process.  Software  engineering 
process  groups  (SEPG)  that  focus  on  improving  the  software  process  evolve 
at  this  level. 

4.  Managed  —  the  organization  has  initiated  comprehensive  software  process 
measurements;  analysis  and  significant  quality  improvements  are  generally 
made.  At  the  managed  level,  statistical  process  control  can  be  applied  to  ana¬ 
lyze  and  improve  the  software  process. 

5.  Optimizing  —  the  organization  now  has  a  foundation  for  continued  improve¬ 
ment  and  optimization  of  the  software  process.  At  the  optimizing  level,  an  or¬ 
ganization  is  in  a  position  to  apply  statistical  process  control  to  maximize  the 
use  of  technology,  increase  productivity,  and  improve  product  quality.  There 
is  also  a  basis  for  quantitative  feedback  on  the  software  process,  so  defect 
prevention  can  take  place. 


1.3.  Maturity  Questionnaire 

The  maturity  questionnaire  [16]  was  designed  in  conjunction  with  the  software  process  ma¬ 
turity  framework.  The  SEI  developed  this  questionnaire  as  a  tool  for  the  assessment  team. 
It  contains  a  structured  set  of  yes/no  questions,  with  each  question  relating  to  a  specific 
software  process  maturity  level  in  the  framework. 
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The  questionnaire  is  used  during  an  assessment  to  do  the  following: 


•  Provide  a  detailed  framework  for  identifying  key  issues  for  discussion  later  in 
the  assessment. 

•  Narrow  the  focus  of  the  assessment  'earn  to  selected  key  areas,  and  help  the 
team  to  focus  on  areas  most  important  to  the  organization. 

•  Identify  areas  about  which  to  request  supporting  information. 

•  Quickly  establish  an  initial  reading  of  an  organization’s  software  process  matur¬ 
ity  level. 

The  maturity  questionnaire  is  used  as  a  "springboard"  to  start  the  assessment  in  the  right 
direction.  The  experience  and  judgment  of  a  trained  assessment  team  determines  the  soft¬ 
ware  process  maturity  level  and  identifies  major  software  issues  of  an  organization.  Thus,  it 
is  not  appropriate  to  use  the  questionnaire  as  a  stand-alone  tool  for  an  assessment. 


1.4.  Contexts  for  Using  the  Maturity  Questionnaire 

There  are  currently  four  contexts  in  which  the  maturity  questionnaire  is  used.  Two  of  these 
involve  assessments: 

•  SEI-assisted  assessments:  software  process  assessments  conducted  by  a 
trained  team  of  software  professionals  from  both  the  SEI  and  the  organization 
being  assessed.  The  scope  of  an  SEI-assisted  assessment  is  usually  a  section 
or  division  of  an  organization  or  possibly  an  entire  site  (one  location  of  an 
organization). 

•  Self-assessment:  software  process  assessments  conducted  at  an  organiza¬ 
tion  by  a  trained  team  of  software  professionals  within  that  organization.  The 
scope  of  the  self-assessment  may  be  a  division  of  an  organization,  an  organiza¬ 
tional  site,  or  the  entire  organization.  Self-assessments  provide  a  way  for  an 
organization  to  examine  its  own  software  process  and  monitor  progress  toward 
software  process  improvement.  The  SEI  trains  in-house  assessment  teams  to 
conduct  self-assessments  of  their  organization. 

In  addition,  the  questionnaire  is  used  in  these  two  contexts: 

•  Software  capability  evaluations:  These  are  evaluations  conducted  as  part  of 
the  Department  of  Defense  software  acquisition  process.  They  may  also  be 
used  by  software  organizations  to  evaluate  subcontractors.  A  software  capa¬ 
bility  evaluation  is  applied  to  site(s)  where  the  proposed  effort  will  be  performed. 

The  method  is  a  subset  of  an  assessment;  an  evaluation  team  concentrates  on 
substantiating  questionnaire  responses.  The  SEI  does  not  perform  evaluations, 
but  it  does  offer  training  in  conducting  software  capability  evaluations. 

•  Assessment  workshops:  At  workshops,  conferences,  tutorials,  and  sym¬ 
posiums,  the  SEI  describes  software  process  assessments,  software  capability 
evaluations,  and  the  assessment  methodology.  These  events  also  enable  the 
SEI  to  receive  feedback  on  the  quality  of  the  maturity  questionnaire  and  to 
gather  industry-profile  data  from  the  questionnaire. 
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The  scope  of  this  report  is  limited  to  the  first  context,  SEI-assisted  assessments.  The  follow¬ 
ing  sections  discuss  assessment  objectives  and  provide  an  overview  and  schedule  for  the 
SEI-assisted  assessment  process. 


1.5.  Assessment  Objectives 

The  main  objectives  of  assessments  are  the  following: 

•  Determine  the  current  state  of  the  software  process  that  is  practiced  on  a  day- 
to-day  basis  in  an  organization. 

•  Identify  key  areas  for  software  process  improvement.  These  are  the  few  high- 
priority  areas  on  which  an  organization  should  focus  improvement  efforts. 

•  Start  the  changes  needed  for  software  process  improvement  within  an  organi¬ 
zation  by  enrolling  key  software  practitioners  and  technical  leaders  in  the 
change  process. 

•  Help  an  organization  focus  on  monitoring  and  improving  its  software  process. 

•  Identify  good  work  (e.g.,  tools,  methods,  techniques)  being  done  by  the  repre¬ 
sentative  projects  so  that  the  organization  as  a  whole  can  benefit. 

1.6.  Overview  of  the  Assessment  Process 

SEI-assisted  assessments  are  typically  conducted  m  six  phases.  Each  assessment  phase 

is  addressed  in  a  separate  chapter  in  this  report  (Chapters  3-8). 

1.  Selection  Phase:  During  the  first  phase,  an  organization  is  identified  as  a 
candidate  for  assessment  (see  Chapter  3).  The  SEI  contacts  the  organization 
to  set  up  an  executive  level-briefing  (see  Section  4.1). 

2.  Commitment  Phase:  In  the  second  phase,  the  organization  commits  to  the 
full  assessment  process  (see  Chapter  4).  An  assessment  agreement  (see  Ap¬ 
pendix  A)  is  signed  by  senior  representatives  of  the  organization  and  the  SEI. 

This  commitment  includes  the  personal  participation  of  the  senior  site  man¬ 
ager,  site  representation  on  the  assessment  team,  and  agreement  to  take  ac¬ 
tion  on  the  assessment  recommendations. 

3.  Preparation  Phase:  The  third  phase  is  devoted  to  preparing  for  the  on-site 
assessment  (see  Chapter  5).  An  assessment  team,  composed  of  members 
from  the  SEI  and  the  organization  being  assessed,  receives  training.  In  addi¬ 
tion,  the  on-site  assessment  is  planned.  The  assessment  participants  are  se¬ 
lected  and  briefed  about  the  assessment  process,  including  times,  duration, 
and  purpose  of  their  participation.  The  questionnaire  can  also  be  filled  out  at 
this  time.  An  example  of  an  assessment  team  training  agenda  is  in  Appendix 
B. 

4.  Assessment  Phase:  In  the  fourth  phase,  *he  on-site  assessment  is  conducted 
(see  Chapter  6).  On  the  first  day,  senior  management  and  assessment  partic¬ 
ipants  are  briefed  as  a  group  about  the  objectives  and  activities  of  the  assess¬ 
ment.  The  project  representatives  complete  the  questionnaire  if  they  have  not 
done  so  previously.  The  resulting  data  and  information  is  reviewed  and 
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analyzed  by  the  assessment  team.  The  team  then  holds  discussions  with 
each  project.  On  the  second  day,  the  team  conducts  discussions  with  the 
functional  area  representatives  or  key  software  practitioners,  who  provide  fur¬ 
ther  insight  into  the  software  process.  Over  the  course  of  the  third  day,  the 
assessment  team  formulates  findings  based  upon  the  information  that  has 
been  collected  on  the  previous  days  ana  gets  feedback  from  the  project 
representatives.  On  the  last  day,  the  findings  are  reviewed  wim  the  project 
representatives  to  help  ensure  that  the  assessment  team  understands  the  is¬ 
sues  correctly.  The  findings  are  revised,  if  necessary,  and  presented  to  the 
assessment  participants  and  senior  site  management.  The  assessme~t  ends 
with  formulation  of  the  recommendations  that  address  the  findings.  An  ex¬ 
ample  of  an  on-sue  assessment  agenda  is  in  Appendix  C. 

5.  Report  Phase:  The  fifth  phase  is  concerned  with  the  final  formulation  and 
communication  of  assessment  findings  and  specific  recommendations  that  ad¬ 
dress  those  findings  (see  Chapter  7).  The  assessment  team  prepares  a  for¬ 
mal  written  report,  which  is  given  to  the  organization  along  with  a  formal  rec¬ 
ommendations  presentation  (see  Section  7.4). 

6.  Assessment  Follow-Up  Phase:  In  the  final  phase,  an  action  team  composed 
entirely  of  professionals  from  the  assessed  organizat.on  is  assembled  and 
charged  with  formulating  an  action  plan  and  facilitating  its  implementation  (see 
Chapter  8).  Typically,  there  is  also  some  continuing  support  and  guidance 
from  the  SEI.  After  approximately  eighteen  months,  a  reassessment  or  self- 
assessment  is  done  by  the  organization  to  determine  progress  a.  id  to  continue 
the  software  process  improvement  cycle. 


1.7.  Schedule  of  Assessment  Events 

Assessments  and  software  process  improvement  are  a  tremendous  amount  of  work.  There 
is  much  to  do  in  a  short  amount  of  time.  While  every  assessment  is  different,  they  follow  the 
same  general  schedule.  After  the  selection  phase,  a  typical  schedule  of  events  looks  like 

t!vs: 


Month  1  —  executive-level  briefing  and  signed  assessment  agreement  between  SEI 
and  the  organization  to  be  assessed 

Month  2  —  half-day  pre-assessment  team  training  presentation  with  readings 

Month  3  —  two-day  assessment  team  training 

Month  4  —  four-day  on-site  assessment 

Month  5  —  report  on  assessment  findings  and  recommendations 

After  receiving  the  final  report,  the  organization  spends  three  to  six  months  developing  an 
action  plan.  Implementation  and  the  software  process  improvement  effort  begin  when  the 
plan  is  approved  by  senior  management.  Eight -en  months  later,  the  organization  may  be 
ready  to  conduct  a  self-assessment  to  evaluate  progress  and  determine  its  new  software 
process  maturity  status. 
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2.  Assessment  Principles 

Assessments  are  a  challenging  activity.  Time  is  short;  organizations  are  complex;  and  the 
software  process  tends  to  be  personnel-intensive  in  nature.  In  order  to  meet  these  chal¬ 
lenges,  the  SEI  has  developed  the  principles  described  in  this  section.  They  are  the  keys  to 
conducting  a  successful  assessment: 

•  Sponsorship 

•  Confidentiality 

•  T  eamwork 

•  Action  orientation 

•  Software  process  framework 


2.1.  Sponsorship 

A  sponsor  is  an  individual  or  group  that  authorizes  the  assessment  and  assumes  final  re¬ 
sponsibility  for  it.  The  sponsor  ensures  that  the  assessment  has  legitimate  backing  and  that 
software  process  improvement  has  official  support  and  financial  guarantees. 

The  sponsor  is  usually  the  senior  site  manager — the  person  who  sets  the  operational  prior¬ 
ities  for  the  organization.  To  be  an  effective  sponsor,  this  person  must  give  high  priority  and 
strong  support  to  the  assessment  and  software  process  improvement.  The  sponsorship  role 
involves  the  following: 

•  Providing  authorization  and  support  for  the  assessment,  a  responsibility  which 
cannot  be  delegated. 

•  Being  visible  and  personally  involved  in  the  assessment  and  follow-up  activities. 

•  Assigning  the  resources  and  qualified  people  for  planning  and  implementing  the 
assessment  and  follow-up  activities. 

•  Building  sustaining  sponsorship  through  middle  management. 

•  Educating  oneself  and  sustaining  sponsors  in  the  assessment  process  and  soft¬ 
ware  process  improvement.  This  understanding  helps  sponsors  make  the  deci¬ 
sions  necessary  to  support  improvement  activities  and  secure  the  critical 
resources  needed. 

•  Agreeing  to  and  signing  the  assessment  agreement. 

2.2.  Confidentiality 

The  power  of  an  assessment  is  that  it  taps  the  knowledge  and  skills  of  the  organization’s 
software  experts  and  project  leaders  (assessment  participants).  Their  accounts  of  the  soft¬ 
ware  process  as  it  is  practiced  at  the  organization  influence  the  results  of  the  assessment. 
Because  assessments  depend  upon  the  honest,  open,  and  accurate  information  that  comes 
from  the  assessment  participants,  it  is  vital  that  they  feel  that  they  can  speak  in  confidence. 
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Confidentiality  is  needed  at  all  organizational  levels.  No  leaks  can  occur — not  to  others  in 
the  organization,  their  bosses,  or  the  organization's  chief  executive.  In  addition,  it  is  impor¬ 
tant  to  note  that  an  assessment  is  not  an  audit  or  mechanism  used  to  single  out  specific 
individuals  or  projects  as  having  unique  problems;  an  assessment  is  done  for  the  benefit  of 
the  organization. 

Senior  management  must  sponsor  and  agree  to  these  notions  of  confidentiality  prior  to  the 
assessment  so  that  team  members  are  not  compromised  by  pressures  to  divulge  confiden¬ 
tial  information.  A  vehicle  used  to  communicate  these  principles  and  help  obtain  the  support 
of  management  is  the  assessment  agreement  (see  Appendix  A). 

Assessments  build  confidentiality  by  several  means: 

•  Composite  assessment  team  findings 

•  individuals  are  not  named 

•  projects  are  treated  as  a  group — five  to  six  projects  are  reviewed  at  one 
time 

•  An  assessment  agreement  that  has  confidentiality  provisions  built  in 

•  An  assessment  team  training  program  and  assessment  presentations  that 
teach  confidentiality  guidelines 

While  many  people  agree  to  confidentiality  in  principle,  it  is  difficult  to  enforce  and  maintain. 
Yet,  if  confidentiality  is  lost,  the  organization  is  at  the  following  risks: 

•  Participants  will  be  less  open  and  provide  less  information. 

•  The  assessment  team  will  have  incomplete  and,  therefore,  less  accurate  infor¬ 
mation  on  which  to  base  its  findings  and  recommendations. 

•  The  assessment  is  likely  to  fail  to  meet  its  goals  and  objectives. 


2.3.  Teamwork 

A  successful  assessment  is  a  collaborative  effort  to  identify  present  good  practice  and  key 
areas  for  improvement.  Teamwork  occurs  on  several  levels:  within  the  assessment  team; 
between  the  team  and  assessment  participants;  and  between  those  involved  in  the  assess¬ 
ment  and  the  rest  of  the  organization. 

One  strength  of  the  assessment  process  is  that  it  is  conducted  as  a  structured  study  by  a 
team  of  knowledgeable  and  experienced  professionals.  Team  members  from  the  organi¬ 
zation  and  the  SE!  each  make  a  valuable  contribution.  The  site  team  members  understand 
the  organization’s  software  process  and  organizational  culture;  and  the  SEI  members  add 
an  independent  professional  viewpoint  to  the  assessment.  The  training  they  receive  togeth¬ 
er,  including  team-building  exercises,  helps  them  to  form  an  effective  and  efficient  assess¬ 
ment  team. 
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Assessments  are  based  on  the  assumption  that  local  software  experts  are  in  the  best  posi¬ 
tion  to  understand  the  organization’s  software  process  issues.  With  the  leading  in-house 
professionals  contributing  their  knowledge  and  skills,  the  assessment  can  be  a  catalyst  to 
motivate  the  entire  organization  to  self-improvement.  Thus,  the  assessment  team  learns 
during  its  training  the  importance  of  listening;  team  members  guard  against  assuming  they 
understand  the  problems  before  participants  fully  discuss  them.  Similarly,  early  briefings  for 
participants  encourage  the  view  of  assessments  as  a  cooperative  effort  focused  on  learning 
and  understanding.  In  addition,  participants  become  aware  that  their  contribution  can  help 
to  change  (and  improve)  the  organization’s  software  process. 

There  must  be  sufficient  rapport  between  the  team  and  the  organization’s  key  people  so  the 
latter  will  share  their  problems,  concerns,  and  creative  ideas.  (See  references  [7],  [23],  and 
[27]  for  related  information.) 


2.4.  Action  Orientation 

Assessments  are  actually  preludes  to  action;  they  are  mechanisms  to  enable  the  beginning 
of  the  change  process.  An  organization  which  chooses  to  participate  in  an  assessment 
must  be  directed  toward  software  process  improvement. 

Assessment  findings  focus  on  identifying  problems  that  are  key  software  process  issues  cur¬ 
rently  facing  the  organization.  Improvement  goals  and  expectations  are  set  as  a  result  of 
conducting  an  assessment.  There  is  a  risk:  if  no  improvement  actions  are  subsequently 
taken,  the  assessment  may  have  a  negative  effect  on  the  current  situation.  In  the  past,  the 
local  practitioners  could  assume  that  management  did  not  entirely  understand  the  issues 
and,  thus,  could  not  be  expected  to  address  them.  After  the  assessment,  this  is  clearly  not 
the  case.  Senior  management  receives  a  written  report  that  describes  the  current  state  of 
the  practice,  identifies  key  areas  needing  improvement,  and  lists  a  set  of  recommendations. 
If  management  does  not  then  take  action,  the  morale  of  the  software  professionals  will  suf¬ 
fer.  An  organization  must  be  prepared  to  take  action,  or  it  should  not  conduct  an  assess¬ 
ment. 


2.5.  Software  Process  Framework 

An  assessment  implies  that  there  exists  a  standard  or  framework  to  measure  against:  an 
organization’s  software  process  needs  to  be  reviewed  in  comparison  with  some  vision  of 
how  software  processes  should  be  performed.  Without  this  standard  or  framework,  an  as¬ 
sessment  can  easily  become  a  loosely  directed  intuitive  exploration.  With  the  framework, 
there  is  a  basis  for  orderly  exploration  as  well  as  a  means  for  establishing  improvement 
priorities.  The  framework  gives  the  team  a  focus  for  working  together  on  the  key  issues  and 
recommendations. 

The  assessment  methodology  described  in  this  report  uses  such  a  framework.  The  SEI 
technical  report,  Characterizing  the  Software  Process:  A  Maturity  Framework  [18],  describes 
the  software  process  maturity  framework  in  detail. 
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3.  Selecting  Candidate  Organizations  for  SEI-Assisted 
Assessments 

To  better  understand  the  criteria  for  selecting  candidate  organizations  for  SEI-assisted  as¬ 
sessments,  it  is  helpful  to  be  aware  of  the  objectives  and  goals  of  the  SEI  for  conducting  the 
assessments.  These  are: 

•  Demonstrate  the  feasibility  and  value  of  conducting  assessments  as  a  means  of 
improving  the  software  process  in  the  DoD  community. 

•  Facilitate  software  process  improvement  in  key  software  organizations. 

•  Increase  SEI  knowledge  of  current  important  software  issues  and  practices. 

The  SEI  performs  approximately  six  SEI-assisted  assessments  each  year.  The  following 
guidelines  are  used  in  making  selection  decisions. 

•  The  SEI  considers: 

1 .  DoD  priorities  and  needs. 

2.  The  level  of  activity  and  affiliation  between  the  SEI  and  the  organization. 

3.  The  extent  to  which  the  assessment  and  the  data  collected  will  be  of 
value  to  the  SEI. 

4.  The  level  of  commitment  of  the  organization’s  management  to  the  terms 
of  the  assessment  agreement  (see  Appendix  A). 

•  The  SEI  avoids  participating  in  assessments  that  involve  software  projects  in 
the  competitive  phase  of  a  procurement. 

•  If  all  criteria  are  equally  satisfied  by  more  than  one  organization,  the  assess¬ 
ments  are  conducted  on  a  first-come,  first-served  basis. 
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4.  Committing  to  the  Assessment 


4.1.  Executive-Level  Briefing 

After  an  organization  has  been  selected  for  an  SEI-assisted  assessment,  a  member  of  the 
SEI  technical  staff  visits  the  organization  to  give  an  executive-level  briefing  to  the  senior  site 
manager  and  his  or  her  immediate  staff.  The  main  topics  of  discussion  for  this  briefing  are: 

•  An  overview  of  the  SEI 

•  The  objectives  and  benefits  of  an  assessment 

•  An  overview  of  software  process  management  and  software  process  improve¬ 
ment  concepts 

•  The  software  process  maturity  framework 

•  An  overview  of  the  assessment  process 

After  the  briefing,  the  organization  can  make  a  commitment  to  an  SEI-assisted  assessment 
by  signing  the  assessment  agreement.  The  next  step  is  to  select  the  assessment  team 
members  and  the  organization's  assessment  coordinator.  The  following  sections  discuss 
the  assessment  agreement  and  assessment  team  selection. 


4.2.  The  Assessment  Agreement 

The  assessment  agreement  is  a  written  agreement  between  the  SEI  and  the  organization 
(for  detailed  terms  and  conditions,  see  Appendix  A).  This  agreement  states  that  senior 
management  will  sponsor  the  assessment  by  providing  the  commitment  and  resources 
necessary — not  only  to  conduct  the  assessment  but  also  to  carry  out  the  assessment  team's 
recommendations  in  an  organizational  action  plan.  The  agreement,  thus,  formally  communi¬ 
cates  to  the  sponsor  the  extent  of  support  needed  to  conduct  and  follow  up  an  assessment. 

In  addition  to  identifying  and  outlining  sponsorship,  the  document  represents  an  agreement 
between  the  organization’s  senior  management,  the  assessment  team,  and  the  assessment 
participants.  It  communicates  specific  information,  such  as: 

•  The  scope  of  the  assessment  (particular  site  or  division) 

•  Assessment  activities,  such  as  SEI  training,  the  participant  briefing,  the  findings 
presentation,  the  final  report,  and  the  action  plan 

•  Confidentiality  provisions 

•  Selection  of  the  assessment  team,  assessment  participants,  and  projects 

The  assessment  agreement  is  signed  by  the  senior  site  manager  and  all  assessment  team 
members. 
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4.3.  Assessment  Team 

Careful  selection  of  the  assessment  team  can  help  ensure  that  the  assessment  is  conducted 
in  a  proper  and  thorough  manner,  that  assessment  principles  are  upheld,  and  assessment 
participants  feel  comfortable  enough  to  contribute  fully  to  the  assessment.  The  following 
sections  discuss  the  roles  and  characteristics  of  assessment  team  members. 

4.3.1.  Assessment  Team  Roles 

In  an  SEI-assisted  assessment,  the  assessment  team  consists  of  an  assessment  team 
leader,  an  SEI  assessment  coordinator,  and  five  to  eight  assessment  team  members — two 
or  three  from  the  organization  and  three  to  five  from  the  SEI.  Experience  has  shown  that 
assessment  teams  of  five  to  ten  members  seem  to  work  well.  Larger  teams  can  be  hard  to 
manage  and  are  generally  less  productive.  A  site  assessment  coordinator  works  with  the 
assessment  team.  The  team  leader  and  SEI  coordinator  always  have  previous  SEI-assisted 
assessment  experience;  and  usually  two  of  the  SEI  members,  including  the  leader,  have 
performed  at  least  two  assessments.  It  is  preferred,  but  not  necessary,  that  the  site  coor¬ 
dinator  also  be  a  member  of  the  assessment  team. 

The  roles  are  as  follows: 

Assessment  team  leader  (member  of  the  SEI  technical  staff)  —  This  person  is  charged 
with  ensuring  that  the  assessment  is  conducted  in  a  proper  and  productive  manner.  He  or 
she  leads  most  of  the  discussions  during  the  assessment  and  also  makes  the  formal  pres¬ 
entations,  including  assessment  team  training  presentations,  the  introductory  management 
presentations  at  the  on-site  assessment,  and  the  assessment  findings  presentation  to  senior 
management.  This  person  should  have  experience  both  in  leading  teams  and  in  making 
public  presentations.  The  leader  works  closely  with  the  SEI  assessment  coordinator. 

SEI  assessment  coordinator  (member  of  the  SEI  technical  staff)  —  The  SEI  assessment 
coordinator  is  in  charge  of  the  assessment  logistics.  Responsibilities  include  assisting  the 
site  assessment  coordinator;  working  with  the  assessment  team  leader;  preparing,  coor¬ 
dinating,  and  conducting  the  assessment  team  training;  and  preparing  for  the  on-site  as¬ 
sessment.  The  SEI  coordinator  also  brings  to  the  assessment  all  materials  required  during 
the  on-site  period. 

Site  assessment  coordinator  (site  professional)  —  The  site  coordinator  is  in  charge  of  all 
the  assessment  logistics  for  the  organization;  this  person  works  with  the  SEI  assessment 
coordinator.  The  site  coordinator  does  not  need  any  previous  assessment  experience. 

Assessment  team  members  —  (site  professionals  and  SEI  technical  staff)  All  team  mem¬ 
bers  attend  the  assessment  team  training  at  the  SEI  and  the  on-site  assessment.  Each 
member  also  writes  a  portion  of  the  assessment  final  report  and  participates  in  a  final  report 
review  at  the  SEI. 
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4.3.2.  Criteria  for  Selecting  the  Assessment  Team 

4.3.2.1.  Essential  characteristics 

Site  assessment  team  members  should  be  all  of  the  following: 

•  Very  knowledgeable  about  the  organization. 

•  Well  respected  within  the  organization. 

•  Motivated  to  improve  the  organization's  software  process. 

•  Willing  to  accept  change  and  have  the  ability  to  help  implement  change  (be  a 
change  advocate  or  change  agent). 

•  Sensitive  to  people  and  able  to  address  questions  to  assessment  participants  in 
a  clear  and  non-threatening  way. 

•  Capable  of  making  a  positive  contribution  as  assessment  team  members. 

Team  members  should  be  opinion  leaders — other  people  listen  to  what  they  have  to 
say — and  team  players  as  well. 

Preferably,  members  should  have  at  least  8-10  years’  experience  as  software  professionals. 

4.3.2.2.  Restrictions 

Site  assessment  team  members  should  not  occupy  positions  within  the  organization  that 
may  lead  to  a  conflict  between  the  assessment  principles  and  their  job  function  (for  ex¬ 
ample,  if  assessment  participants  believe  that  information  they  volunteer  has  the  potential  to 
affect  them  adversely  after  the  assessment,  they  may  not  speak  freely  or  not  speak  at  all). 
Nor  should  there  be  a  conflict  between  their  function  on  the  assessment  team  and  their  reg¬ 
ular  job  function.  To  help  ensure  a  free  flow  of  information,  organization  assessment  team 
members  should  not  hold  positions  that  involve  any  of  the  following  activities: 

•  Acting  as  manager  of  a  project  included  in  the  assessment,  nor  the  manager  of 
such  a  person. 

•  Working  on  or  being  directly  involved  with  reviewing  or  supporting  a  project 
being  assessed. 

•  Currently  serving  in  a  software  audit  or  software  quality  assurance  (SQA)  posi¬ 
tion  for  any  of  the  projects  being  reviewed. 

4.3.3.  Time  Commitments  of  Team  Members 

Dedicated  participation  is  essential  during  assessment  team  training,  the  on-site  assess¬ 
ment,  and  the  review  of  the  final  report.  During  these  meetings,  phone  calls  are  held,  all 
other  meetings  and  commitments  are  rescheduled,  and  the  members  are  required  to  be  on 
time  to  every  assessment  session.  Elimination  of  outside  interruptions  allows  the  assess¬ 
ment  to  proceed  smoothly. 
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Listed  below  are  estimates  of  the  time  a  team  member  will  need  for  assessment  activities. 

•  Pre-assessment  team  training  presentation  for  all  team  members,  held  on-site 
(see  Section  5.1):  one-half  day. 

•  Review  of  reading  materials  provided  at  the  pre-assessment  team  training  pres¬ 
entation  (see  Section  5.1):  several  days. 

•  Assessment  team  training  session  for  all  team  members,  held  at  the  SEI  (see 
Section  5.2):  two  days,  plus  travel  and  preparation  time. 

•  Preparation  by  the  site  assessment  coordinator  for  the  on-site  assessment  (see 
Chapter  5):  25%-30%  of  that  person's  time  for  about  a  month — from  the  end  of 
the  assessment  team  training  to  the  beginning  of  the  on-site  assessment. 

•  The  on-site  assessment,  which  involves  all  team  members  (see  Chapter  6): 
four  full  days. 

•  Participation  of  each  team  member  in  writing  a  portion  of  the  final  report  during 
the  week  following  the  assessment  (see  Chapter  7):  two  to  three  days. 

•  Final  report  review  by  all  team  members,  held  at  the  SEI  approximately  three 
weeks  after  the  assessment  (see  Section  7.2):  one  day.  plus  travel  and  prepa¬ 
ration  time. 

After  the  assessment,  the  site  team  members  may  become  members  of  an  action  plan  team 
(see  Chapter  8)  or  a  software  engineering  process  group  (see  Section  8.4). 
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5.  Preparing  for  the  Assessment 

This  chapter  discusses  the  pre-assessment  activities  that  set  the  stage  for  a  smooth  and 
orderly  on-site  assessment. 


5.1.  Pre-Assessment  Team  Training 

After  the  organization  has  committed  to  the  assessment  and  has  selected  its  assessment 
team  members,  the  SEI  assessment  team  leader  or  the  coordinator  visit  the  organization  to 
give  a  half-day  presentation  to  the  site  team  members.  The  main  objectives  of  the  pre¬ 
assessment  training  presentation  are:  set  realistic  expectations  and  accurately  describe  the 
commitments  needed  for  the  assessment:  answer  questions;  and  hand  out  technical  papers 
and  other  information.  The  presenter  brings  an  advance  preparation  notebook  for  each  site 
assessment  team  member.  The  notebooks  include  the  following: 

•  SEI  overview.  This  section  is  an  overview  of  the  SEI  mission  and  technical  pro¬ 
grams. 

•  Readings  on  software  process  management. 

•  Assessment  process  overview  (see  Section  5.2.1). 

•  SEI-assisted  assessment  preparation.  This  section  includes  an  overview  of  the 
advance  preparation  materials,  the  organizational  briefing,  information  about 
project  selection,  and  the  assessment  schedules. 

•  Maturity  questionnaire  overview  (see  Section  1.3).  This  section  includes  the 
report,  A  Method  for  Assessing  the  Software  Engineering  Capability  of 
Contractors  [  16]. 

•  Readings  on  managing  technological  change. 

At  the  end  of  the  presentation,  the  organization's  team  members  know  what  they  need  to  do 
before  coming  to  the  SEI  for  training: 

•  Read  the  material  in  the  advance  preparation  notebook. 

•  Identify  a  number  of  candidate  projects,  a  subset  of  which  will  be  selected  by 
the  team  to  participate  in  the  assessment. 

•  Prepare  an  organizational  briefing  to  be  given  at  the  assessment  team  training. 

5.1.1.  Project  Selection  Criteria 

After  the  pre-assessment  team  training  presentation,  the  organization  (usually  the  site  as¬ 
sessment  team  members,  with  the  approval  of  management)  chooses  10  to  12  key  projects 
as  candidates  for  the  assessment.  During  the  on-site  planning  portion  of  the  training,  the 
assessment  team  as  a  group  selects  5  or  6  of  these  projects  along  with  2  others  as  backup 
projects.  Backup  projects  are  needed  in  case  one  of  the  projects  selected  cannot  partic¬ 
ipate  in  the  assessment. 
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The  projects  should  be  selected  using  the  criteria  listed  below. 

•  The  project  should  be  representative  of  the  software  of  most  concern  to  the 
organization. 

•  The  project  should  be  representative  of  the  software  process  used  in  the  organ¬ 
ization  as  a  whole. 

•  The  project  should  have  a  staff  of  at  least  four  members  and  a  life  cycle  span  of 
at  least  six  months. 

•  The  project  manager  should  support  participation  in  the  assessment.  (The  Droj 
ect  manager  should  not  be  forced  to  participate.) 

•  The  projects  selected  should  be  in  varying  software  development  phases  (such 
as  requirements,  design,  implementation,  testing,  maintenance).  In  other 
words,  the  projects  should  not  all  be  in  the  same  phase. 

•  There  should  be  a  mix  of  projects  of  varying  duration  and  size. 

•  The  projects  must  not  be  in  the  competitive  phase  of  a  procurement  process. 

•  A  project  should  not  be  included  if  an  assessment  team  member  participates  in 
or  manages  it. 

An  important  issue  to  remember  is  that  the  findings  of  an  assessment  are  composite  across 
the  projects.  Choosing  a  project  whose  process  differs  drastically  from  the  others  may 
make  it  difficult  to  generate  composite  findings.  This  might  occur  if  the  projects  are  from 
different  sections  or  divisions  whose  organizational  structures  dictate  different  software  de¬ 
velopment  processes. 

5.1.2.  Organizational  Briefing 

The  site  assessment  coordinator,  with  the  help  of  other  site  team  members,  presents  infor¬ 
mation  about  the  organization  at  the  assessment  team  training.  The  purpose  of  this  briefing 
is  to  give  SEI  team  members  a  good  understanding  of  the  organization  and  the  types  of 
software  it  develops.  Typical  briefing  information  includes: 

•  A  recent  organizational  chart.  SQA,  tool  groups,  metrics,  software  engineering 
process  groups,  etc.,  should  be  pointed  out.  The  projects  being  considered 
should  also  be  pointed  out,  with  an  explanation  of  how  they  fit  into  the  organi¬ 
zation. 

•  The  types  of  software  systems  developed  or  maintained. 

•  The  total  number  of  software  professionals  in  the  organization. 

•  Any  internal  software  policies/standards. 

•  Information  about  the  software  engineering  environment  and  organizational  cul¬ 
ture. 
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5.2.  Assessment  Team  Training  Overview 

All  assessment  team  members  participate  in  a  two-day  assessment  team  training  program 
at  the  SEI,  not  only  to  familiarize  them  with  tne  assessment  activities,  but  also  to  build  a 
cohesive  team.  The  assessment  team  leader  and  the  SEI  coordinator  conduct  the  training 
session,  which  is  discussed  in  the  next  section. 

The  training  allows  the  team  members  to  do  the  following: 

•  Walk  through  the  assessment  process. 

•  Simulate  parts  of  an  actual  assessment  and  obtain  some  hands-on  experience. 

•  Fill  out  the  maturity  questionnaire  and  perform  some  data  analysis. 

•  Participate  in  team-building  exercises. 

•  Learn  or  review  software  process  management  concepts. 

Another  important  activity  that  takes  place  during  the  training  period  is  planning  for  the  on¬ 
site  assessment.  Team  members  learn  what  needs  to  be  done  next,  the  purpose  behind  it, 
their  role,  and  the  dates  and  times  of  assessment  activities.  There  is  also  some  plann'ng  for 
assessment  follow-up  activities. 

5.2.1.  Assessment  Team  Training 

A  typical  assessment  team  training  session  takes  about  two  days  (see  Appendix  3).  During 
that  time,  the  team  is  involved  in  the  following  presentations,  meetings,  and  exercises: 

•  Process  management  overview  —  Material  dealing  with  process  manage¬ 
ment  is  introduced  in  this  session.  Process  management  premises  and  prin¬ 
ciples  are  discussed,  along  with  the  software  process  maturity  framework. 
Finally,  these  concepts  and  framework  are  used  to  characterize  the  software 
improvement  process. 

•  Organizational  briefing  —  See  Section  5.1.2 

•  Assessment  process  overview  —  The  purpose  and  objectives  of  assess¬ 
ments  are  discussed,  along  with  their  role  in  software  process  improvement. 

The  assessment  methodology  is  discussed  in  detail. 

•  Maturity  questionnaire  —  This  session  describes  the  maturity  questionnaire, 
its  role  in  the  assessment,  and  how  it  is  used.  To  obtain  hands-on  experience 
in  using  it,  team  members  fill  out  the  questionnaire  on  a  software  project  with 
which  they  are  familiar. 

•  Review  of  project  responses  —  During  this  sess'on,  the  team  members  simu¬ 
late  some  of  the  activities  in  an  assessment.  The  data  from  the  questionnaire 
forms  are  reviewed  to  identify  similarities,  differences,  misunderstandings,  and 
anomalies.  "Probe  areas'  are  then  identified,  where  supporting  materials  would 
be  requested  in  an  actual  assessment. 

•  Assessment  discussions  —  This  presentation  describes  the  role  and  objec¬ 
tives  of  the  assessment  discussions,  and  the  methods  used  for  conducting 
them.  Functional  area  discussions  and  project  discussions  take  place  between 
the  assessment  team  and  the  assessment  participants. 
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•  Team-building  exercises  —  These  exercises  help  the  team  members  learn  to 
work  together  effectively. 

•  Assessment  findings  —  This  presentation  explains  the  role  of  the  findings  and 
the  procedure  used  for  their  formulation.  Some  helpful  guidelines  are  provided; 
the  structure  of  the  findings  is  discussed;  and  some  sample  findings  are  re¬ 
viewed.  The  importance  of  the  on-site  findings  presentation  is  also  discussed. 

•  Final  report  preparation  —  This  presentation  discusses  the  final  report  of  the 
assessment  team's  findings  and  recommendations.  The  report  format  is  de¬ 
scribed  in  detail,  as  well  as  the  structure  of  the  findings  and  recommendations. 

The  entire  assessment  team  is  involved  in  the  creation  and  review  of  the  final 
report. 

•  Planning  for  the  assessment  —  Plans  for  the  on-site  assessment  are  dis¬ 
cussed  in  detail.  These  include  developing  a  detailed  on-site  schedule  and 
daily  agenda  (see  Appendix  C),  selecting  the  specific  projects  to  be  examined 
and  the  backup  projects,  and  identifying  the  people  to  be  interviewed  and  the 
facilities  required.  Since  the  people  and  facilities  in  most  software  organiza¬ 
tions  are  heavily  committed,  significant  advance  notice  is  generally  recom¬ 
mended. 

At  the  end  of  the  training,  team  members  should  be  ready  to  participate  in  the  upcoming 
assessment.  But  there  is  still  much  work  to  be  done  in  preparation  for  the  assessment,  as 
discussed  in  the  next  sections. 


5.3.  Preparing  for  the  On-Site  Period 

Much  of  the  preparation  for  the  on-site  period  is  done  by  the  site  assessment  coordinator, 
but  all  site  team  members  are  encouraged  to  participate  since  there  is  much  work  to  be 
done.  The  following  sections  address  the  major  tasks  that  should  be  performed  a  week  or 
two  before  the  on-site  assessment: 

•  Selecting  functional  area  representatives. 

•  Briefing  the  assessment  participants  to  prepare  them  for  the  assessment. 

•  Filling  out  the  questionnaire  for  each  project  (before  or  during  the  assessment). 

5.3.1.  Criteria  for  Selecting  Functional  Area  Representatives 

Functional  area  representatives  are  software  practitioners  from  specific  software  process 
areas.  Typically,  there  are  five  groups  with  four  to  six  representatives  each.  Listed  below  is 
an  example  of  how  functional  areas  can  be  grouped  for  an  assessment: 

•  Software  quality  assurance  and  release 

•  Software  integration  and  test 

•  Coding  and  unit  test 

•  Detailed  design 

•  Requirements  and  high-level  design 
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The  role  of  the  functional  area  representatives  is  to  balance  the  management  view 
presented  by  the  project  leaders  with  the  technical  view  of  software  practitioners.  The 
representatives  participate  in  informal,  unstructured  discussions  that  allow  them  to  express 
their  professional  views  about  issues  and  problems  facing  the  organization.  Functional  area 
representatives  should  be  actually  working  on  software  projects;  they  should  not  be  staff 
personnel  or  managers.  They  should  be  widely  recognized  within  the  organization  as  tech¬ 
nical  experts  in  a  functional  area.  Representatives  should  also  be  opinion  leaders  among 
their  peers  and  should  fully  understand  the  confidentiality  provisions. 

5.3.2.  Briefing  the  Assessment  Participants 

At  least  one  week  before  the  on-site  period,  all  assessment  participants  should  be  briefed  by 
the  site  assessment  coordinator  on  the  following: 

•  Assessment  principles. 

•  Overview  of  the  assessment. 

•  The  detailed  schedule  for  the  on-site  period,  outlining  the  purpose,  location, 
time,  and  length  of  each  meeting,  along  with  identifying  who  should  attend  (see 
Appendix  C). 

The  pre-assessment  briefing  informs  participants  about  the  assessment  process  and  en¬ 
ables  them  to  plan  for  their  involvement  in  the  assessment,  including  giving  their  managers 
and  colleagues  an  idea  of  the  extent  of  their  participation. 

5.3.3.  Filling  Out  the  Maturity  Questionnaire 

The  maturity  questionnaire  is  filled  out  by  project  representatives  either  before  or  during  the 
assessment.  Instructions  that  describe  the  questionnaire  and  the  procedures  for  completing 
it  are  given  to  the  site  team  members  during  assessment  team  training.  Filling  out  the  ques¬ 
tionnaire  beforehand  saves  some  valuable  time  during  the  assessment.  However, 
participants’  questions  will  need  to  be  answered,  and  this  can  only  be  done  by  a  trained 
assessment  team  member. 
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6.  Conducting  the  Assessment 

This  chapter  discusses  the  sequence  of  activities  constituting  the  on-site  portion  of  the  as¬ 
sessment.  Typically,  these  activities  require  four  days  to  complete. 


6.1.  Introductory  Management  Meetings 

The  introductory  management  meetings  include  the  assessment  overview  and  the  assess¬ 
ment  participant  briefing. 

Assessment  Overview 

The  assessment  begins  with  presentations  to  the  senior  site  manager,  his  or  her  immediate 
staff,  and  the  assessment  participants.  The  main  objective  of  this  presentation  is  to  provide 
an  overview  of  the  assessment  process  and  describe  its  relationship  to  software  process 
improvement  and  software  process  management.  It  is  vital  that  senior  management  attend 
this  meeting  to  show  their  commitment  to  the  assessment  and  their  support  for  software 
process  improvement  (see  Section  2.1). 

Assessment  Participant  Briefing 

After  the  assessment  overview  meeting,  assessment  participants  receive  further  information 
about  the  assessment,  including  the  schedule.  The  objectives  of  this  meeting  are  to  ensure 
that  the  participants’  questions  are  answered  and  to  confirm  that  they  understand  their  role 
in  the  assessment— where  they  should  be,  when  they  should  be  there,  and  what  is  expected 
of  them. 

6.2.  Questionnaire  Data  Analysis  Session 

As  mentioned  earlier,  the  questionnaire  can  be  filled  out  either  before  or  during  the  assess¬ 
ment.  After  the  questionnaires  have  been  completed,  the  assessment  team  meets  in  a 
private  session  to  analyze  the  responses  and  determine  the  initial  maturity  level  of  the  or¬ 
ganization.  Team  members  analyze  questionnaire  data  for  similarities,  differences, 
anomalies,  and  misunderstandings;  and  they  identify  areas  where  questions  need  to  be 
asked  and  supporting  materials  need  to  be  requested.  The  session  is  private  because  con¬ 
fidentiality  has  been  guaranteed  to  both  the  assessment  participants  and  the  individual  proj¬ 
ects. 

Supporting  materials  are  important  for  verifying  that  a  particular  question  has  been  an¬ 
swered  accurately.  What  is  important  here  is  verifying  the  actual  practice  in  an  organization, 
that  is,  the  software  process  that  is  used  on  a  daily  basis.  The  assessment  team  is  inter¬ 
ested  in  what  is  actually  being  done,  not  what  may  be  written  in  a  set  of  manuals.  For 
example,  one  question  on  the  questionnaire  may  be  "Are  internal  software  design  reviews 
conducted?"  The  supporting  material  requested  may  be  the  review  action  items  or  minutes 
of  any  design  reviews  conducted  on  the  project. 
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Assessments,  by  their  nature,  tend  to  be  negative  because  they  uncover  the  most  important 
software  process  issues  or  problems  facing  an  organization.  But  the  findings  of  an  assess¬ 
ment  can  also  be  positive,  for  another  important  objective  of  this  session  is  to  identify  areas 
where  good  work  is  being  done.  Finding  examples  of  good  work  and  making  them  visible 
benefits  the  entire  organization. 


6.3.  Project  Discussions,  Session  I 

After  the  questionnaire  data  has  been  analyzed,  the  assessment  team  meets  with  the  proj¬ 
ect  representatives,  one  project  at  a  time.  Meeting  on  a  project-by-project  basis  maintains 
confidentiality  among  projects.  The  purpose  of  this  session  is  to  allow  the  team  to  get  a 
sense  of  whether  the  questionnaire  was  filled  out  accurately  and  whether  the  maturity  ievel 
of  the  projects  seems  accurate,  to  verify  project  differences  and  similarities,  and  to  request 
supporting  materials.  The  assessment  team  members  listen  to  the  representatives,  asking 
questions  about  how  the  project  does  its  day-to-day  work  of  developing  and  maintaining 
software  for  the  project.  For  example,  the  assessment  team  asks  about  configuration  man¬ 
agement,  project  management,  and  quality  control. 

Assessment  Team  Wrap-Up  Meeting 

After  the  project  discussions,  the  assessment  team  meets  alone  to  discuss  the  progress  of 
the  assessment,  review  initial  findings,  and  prepare  for  the  functional  area  discussions. 


6.4.  Functional  Area  Discussions 

On  the  second  day  of  the  assessment,  the  team  conducts  discussions  with  software  profes¬ 
sionals  from  each  major  functional  area  of  the  organization's  software  process.  The  overall 
objective  for  this  session  is  to  enable  the  assessment  team  to  learn  what  the  software  prac¬ 
titioners  (non-management)  consider  the  most  important  software  engineering  issues  facing 
their  functional  area  or  the  organization  as  a  whole. 

The  discussions  are  informal  sessions  in  which  assessment  participants  are  asked  to  de¬ 
scribe  how  they  do  their  work.  These  sessions  allow  the  assessment  team  to  learn  the 
details  of  the  actual  day-to-day  software  practice.  Managers  do  not  attend  these  discus¬ 
sions  so  that  software  practitioners  will  feel  more  comfortable  about  speaking  freely. 

Near  the  end  of  the  session,  the  assessment  team  leader  asks  each  functional  area  repre¬ 
sentative  a  key  question:  "If  you  could  change  one  thing  to  improve  the  quality  or  produc¬ 
tivity  of  your  work,  what  would  it  be?"  The  answers  to  this  question  are  important  because 
experience  has  shown  that  the  people  working  directly  with  the  product  (software,  documen¬ 
tation,  etc.)  generally  have  many  good  ideas  for  improvement. 
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Assessment  Team  Wrap-Up  Meeting 

To  conclude  day  two,  the  assessment  team  holds  a  wrap-up  meeting  to  review  the  day's 
findings,  contrast  and  compare  them  to  the  previous  day’s  findings,  and  prepare  for  the  next 
day  of  the  assessment.  Considerable  time  is  spent  in  this  session  in  reaching  team  consen¬ 
sus  on  the  preliminary  findings. 


6.5.  Project  Discussions,  Session  II 

To  begin  the  third  day  of  the  assessment,  the  assessment  team  again  meets  with  the  project 
representatives,  one  project  at  a  time.  The  purpose  of  these  meetings  is  to  discuss  any 
open  issues,  to  review  the  requested  supporting  materials,  and  to  discuss  the  assessment 
team’s  preliminary  findings  with  the  project  representatives.  During  the  meeting,  more  is¬ 
sues  may  come  up  and  further  discussion  may  be  needed.  The  intent  here  is  to  gather 
additional  data  to  support  and  refine  the  findings,  and  to  identify  any  other  problems  or  is¬ 
sues. 

During  these  discussions,  the  team  reviews  examples  of  good  work  that  have  been  provided 
by  the  project  participants.  During  the  final  findings  presentation,  there  is  time  set  aside  to 
describe  good  work  found  during  the  assessment. 


6.6.  Final  Findings  Formulation 

After  the  project  discussions  are  completed,  the  assessment  team  meets  to  formulate  the 
findings  and  prepare  the  findings  presentation.  The  findings  represent  the  assessment 
team’s  view  of  the  most  important  software  process  issues  that  currently  face  the  organi¬ 
zation.  The  findings  are  based  on  the  maturity  questionnaire  responses,  discussions  with 
the  assessment  participants,  and  discussions  among  the  assessment  team  members.  The 
findings  represent  the  starting  point  for  formulating  recommendations  (see  Section  6.9). 

Extended  and  intense  discussions  are  often  required  to  reach  a  team  consensus  on  the 
findings.  Since  there  is  no  time  scheduled  the  next  day  for  continuing  these  discussions, 
this  meeting  often  continues  into  the  evening.  (This  is  where  the  team-building  exercises 
during  the  assessment  team  training  can  pay  off.)  Once  the  findings  have  been  formulated, 
the  team  then  prepares  the  findings  presentation. 

At  the  conclusion  of  the  third  day,  the  assessment  team  should  have  achieved  a  team  con¬ 
sensus  for  the  assessment  findings  and  completed  the  preliminary  findings  presentation. 


CMU/SEI-89-TR-7 


25 


6.7.  Assessment  Findings  Presentation 

The  findings  presentation  includes  the  following: 

•  Scope  of  the  assessment  —  This  includes  the  site  or  division  assessed,  the 
names  of  the  projects  assessed,  and  the  names  of  the  functional  areas  that 
were  interviewed. 

•  Conduct  of  the  assessment  —  This  is  a  high-level  discussion  of  the  assess¬ 
ment,  and  how  it  went  in  general.  All  assessment  participants  and  support  staff 
are  thanked  for  their  time,  cooperation,  and  assistance. 

•  Composite  organizational  status  —  The  software  process  maturity  level  of 
the  organization  is  noted  here.  It  is  emphasized  that  the  success  of  an  organi¬ 
zation  is  based  on  the  action  taken  in  response  to  recommendations,  and  not 
on  a  score. 

•  Strengths  —  Any  organizational  strengths  are  noted  here  (e.g.,  examples  of 
good  project  work). 

•  Findings  —  Findings  are  summarized,  and  each  finding  is  then  discussed  in 
detail.  The  consequences  of  each  finding  are  pointed  out,  and  general  ex¬ 
amples  of  the  findings  are  used  if  appropriate. 

•  Next  steps  —  The  steps  following  an  assessment  are  discussed:  the  recom¬ 
mendations  formulation,  the  final  report,  and  the  action  plan,  along  with  an  an¬ 
ticipated  schedule. 

Assessment  Team  Session 

On  the  last  day  of  the  assessment,  the  assessment  team  leader  presents  the  findings  pres¬ 
entation  to  the  assessment  team.  The  purpose  of  this  session  is  to  catch  any  errors  and 
fine-tune  the  presentation. 

Project  Composite  Feedback 

Next,  the  assessment  team  leader  gives  a  dry  run  of  the  findings  presentation  to  all  the 
project  representatives  together,  along  with  the  assessment  team.  The  purpose  of  this 
meeting  is  to  hear  any  concerns  from  the  project  representatives,  get  early  feedback  before 
the  final  findings  presentation  to  senior  management,  and  answer  any  questions. 

Final  Assessment  Findings  Presentation 

Finally,  the  assessment  team  leader  gives  the  final  assessment  findings  presentation  to  the 
senior  site  manager,  his  or  her  immediate  staff,  and  the  assessment  participants.  The  pur¬ 
pose  of  this  session  is  to  present  the  assessment  team’s  view  of  the  most  important  soft¬ 
ware  process  issues  facing  the  organization.  Because  the  organization’s  findings  are  a 
composite,  confidentiality  is  ensured. 
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6.8.  Senior  Management  Meeting 

After  the  final  findings  presentation,  the  senior  site  manager,  immediate  staff  (if  desired), 
and  the  assessment  team  leader  hold  an  executive  meeting  to  discuss  next  steps.  The 
purpose  of  this  meeting  is  to  confirm  the  time  for  the  recommendations  presentation  and 
final  report,  discuss  the  importance  of  forming  the  action  plan  team  and  developing  the  ac¬ 
tion  plan,  and  address  any  questions  or  concerns  of  management. 


6.9.  Recommendations  Formulation 

After  the  assessment  findings  presentation  and  the  senior  management  meeting,  there  is 
one  more  on-site  session:  the  initial  formulation  of  the  recommendations  to  address  the 
findings.  The  purpose  of  this  meeting  is  to  obtain  a  team  consensus  on  the  recommen¬ 
dations  to  be  documented  in  the  final  report  (see  Chapter  7)  and  to  assign  a  portion  of  that 
report  to  each  assessment  team  member. 

Some  guidelines  used  by  the  team  for  formulating  the  recommendations  follow: 

•  Address  each  key  finding,  though  there  need  not  be  a  one-to-one  correspon¬ 
dence  between  findings  and  recommendations. 

•  Limit  the  number  of  recommendations. 

•  Make  the  recommendations  specific  and  concise. 

•  Prioritize  the  recommendations. 

•  Focus  on  what  the  recommendations  are,  not  how  they  will  be  implemented. 

•  Be  sensitive  to  how  the  recommendations  affect  organizational  resources. 
Recommendations  should  be  realistic  enough  to  be  accomplished. 

In  forming  recommendations,  the  team  should  be  sensitive  to  the  impact  on  organization 
resources.  Experience  indicates  that  organizations  can  only  handle  a  few  high-priority  is¬ 
sues  at  a  time.  For  example,  working  on  20  high-priority  recommendations  at  once  is  unre¬ 
alistic.  However,  one  effective  solution  is  to  put  the  recommendations  into  "priority  groups." 
For  example,  if  there  are  10  recommendations,  the  highest  priority  recommendations  (say  3) 
are  grouped  together  into  priority  group  one,  which  will  be  addressed  first.  Then  the  other 
recommendations  are  grouped  as  well,  with  each  group  having  a  priority.  With  ordered 
groupings,  the  action  plan  team  has  a  stronger  direction  for  planning  and  implementation. 

At  the  conclusion  of  this  meeting,  the  next  steps  are  planned  for  the  final  report.  The  times, 
dates,  and  places  for  the  final  report  review  and  for  the  recommendations  presentation  are 
determined.  This  is  planning  is  necessary  because  there  is  a  tremendous  amount  of  work  to 
be  done  in  producing  the  final  report  (see  next  chapter). 
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7.  Assessment  Final  Report 


7.1.  Overview 

Following  the  on-site  assessment  phase,  the  assessment  team  prepares  a  final  report  of  the 
assessment  findings  and  recommendations.  The  purpose  of  the  final  report  is  to  document 
the  assessment,  describe  the  key  findings,  and  make  specific  recommendations  to  the  or¬ 
ganization  based  on  those  findings.  The  entire  assessment  team  contributes  to  the  creation 
and  review  of  the  final  report. 

A  carefully  written  final  report  is  vital  since  it  is  the  only  permanent  record  of  the  assess¬ 
ment.  It  is  also  used  as  a  reference  for  the  action  plan  formulation  and  execution,  and  for 
future  reassessments. 

A  typical  schedule  for  report  preparation  is  as  follows: 

1  st  week  after  assessment:  Each  team  member’s  portion  of  the  report  is  due. 

2nd  week:  Initial  report  review  at  SEI. 

3rd  week:  Final  report  review  a*  SEI. 

4th  week:  Final  report  submitted  to  SEI  Information  Management  for  editing. 

5th  or  6th  week:  Recommendations  presentation  at  the  organization. 


7.2.  Reviews 

Experience  has  shown  that  final  report  content  and  quality  are  best  improved  through  a  doc¬ 
ument  review  process.  Typically,  there  are  two  report  reviews  at  the  SEI:  an  initial  review  by 
SEI  assessment  team  members,  and  a  final  review  by  the  entire  assessment  team.  Two 
reviews  are  generally  adequate.  The  organization’s  assessment  team  members  are  invited 
to  the  initial  review,  but  it  is  often  difficult  (or  too  costly)  for  them  to  travel  to  the  SEI  twice  in 
one  month,  especially  immediately  after  the  assessment. 

The  initial  review  is  an  informal,  half-day  review.  For  the  final  review,  the  organization's 
assessment  team  members  visit  the  SEI  for  one  full  day.  This  review  is  more  formal  since 
the  group  is  larger  and  this  is  the  last  review  before  the  report  is  completed.  The  overall 
goals  of  the  final  review  are  to  reach  team  consensus  on  the  final  report  and  to  ensure  that  it 
is  accurate  and  of  high  quality. 
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7.3.  Report  Structure 

The  final  report  contains  the  following  sections: 

1.  Executive  Summary  —  a  background  of  the  visit,  an  acknowledgment  of  the 
organization's  efforts,  and  a  summary  of  recommendations  in  order  of  priority 
(or  priority  groups). 

2.  Organizational  Status  —  a  brief  statement  of  the  software  process  maturity 
level  that  was  determined  by  the  assessment  team,  a  brief  summary  of  or¬ 
ganizational  strengths  and  good  work  observed,  and  planned  improvements 
that  were  noted. 

3.  Key  Findings  —  the  key  findings  of  the  assessment  team. 

4.  Recommendations  —  the  assessment  team's  recommendations. 

5.  Appendices 

•  Assessment  Agreement  —  a  copy  of  the  signed  assessment  agree¬ 
ment. 

•  Assessment  Conduct  —  a  brief  overview  of  assessment  methodology 
and  process  maturity  model,  a  list  of  the  assessment  team  members,  a 
list  of  the  projects  included  in  the  assessment,  and  a  chronology  of  key 
events. 

•  Cross-Reference  —  a  cross-reference  of  key  findings  and  recommen¬ 
dations  (this  is  useful  since  findings  and  recommendations  do  not  al¬ 
ways  correspond  one-to-one). 


7.4.  Recommendations  Presentation 

After  the  final  report  is  completed,  the  assessment  team  leader  gives  a  recommendations 
presentation  at  the  organization  to  the  senior  site  manager,  his  or  her  immediate  staff,  the 
organization's  assessment  team  members,  and  the  assessment  participants.  Typically,  the 
SEI  assessment  coordinator  and  assessment  team  members  do  not  return  for  this  presen¬ 
tation.  The  main  purpose  of  this  presentation  is  to  provide  an  overview  of  the  recommen¬ 
dations  to  senior  management  and  to  deliver  the  final  report. 

The  recommendations  presentation  usually  provides  a  brief  overview  of  the  software  proc¬ 
ess  maturity  framework  and  the  assessment,  summarizes  the  findings,  and  presents  con¬ 
cise  statements  of  the  recommendations  along  with  their  benefits.  In  addition,  next  steps 
are  discussed:  the  action  plan  formulation,  the  action  plan  review  with  SEI,  and  a  reassess¬ 
ment  in  approximately  eighteen  months. 
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8.  Post-Assessment  Follow-Up 

Although  the  goal  of  an  assessment  is  to  accurately  characterize  the  current  state -of  the 
software  process  that  is  practiced  in  an  organization  and  identify  the  key  software  issues, 
the  ultimate  intent  is  to  be  the  catalyst  for  software  process  improvement  in  the  assessed 
organization.  This  chapter  focuses  on  post-assessment  activities  that  are  effective  in  in¬ 
creasing  the  likelihood  that  software  improvements  will  occur. 


8.1.  Action  Plan  Formulation 

Following  the  on-site  recommendations  presentation,  the  assessed  organization  prepares 
an  action  plan  that  specifies  how,  when,  and  by  whom  each  recommendation  is  to  be  imple¬ 
mented.  Each  action  that  addresses  a  recommendation  is  called  an  action  task.  The  pur¬ 
pose  of  the  action  plan  is  to  identify  and  plan  the  work  effort  for  each  action  task,  identify 
responsibility  and  resources,  and  provide  a  mechanism  for  tracking  and  reporting  on  task 
status.  The  organizational  units  or  projects  implementing  the  action  task  participate  in  its 
development  and  review,  thus  building  a  commitment  to  the  change  effort. 

Sponsorship  and  commitment  from  senior  management  is  critical  after  the  assessment.  In 
order  for  the  action  tasks  to  be  successful,  management  must  sponsor  the  action  plan  for¬ 
mulation,  review,  and  implementation.  Ideally,  the  senior  site  manager  names  a  site  as¬ 
sessment  team  member  to  guide  the  effort  to  develop  the  action  plan.  This  person  should 
be  fully  knowledgeable  of  the  issues  to  be  addressed;  and  he  or  she  should  be  respected  by 
the  people  who  will  be  involved.  It  is  a  good  idea  to  involve  other  leading  professionals  from 
the  projects  in  action  plan  development;  this  not  only  contributes  to  the  quality  of  the  results, 
but  it  also  facilitates  acceptance  of  the  action  plan.  Experience  indicates  that  action  plan 
development  is  most  effective  when  done  as  a  team  effort  involving  all  organizational  units 
involved  in  implementation. 

An  action  plan  should  be  sufficiently  detailed  to  provide  a  clear  guide  for  execution;  it  speci¬ 
fies  the  following: 

•  What  recommendations  will  be  implemented  (action  tasks). 

•  What  recommendations  will  not  be  implemented,  and  why. 

•  How  each  action  task  will  be  implemented. 

•  Resources  (people  and  dollars)  required  to  implement  each  action  task. 

•  The  person  or  people  responsible  for  each  action  task. 

•  Action  task  schedules,  with  appropriate  management  review  checkpoints. 

The  action  plan  should  identify  quarterly  checkpoints  for  periodic  management  review  (see 
Section  8.3).  Only  a  few  (one  to  three)  action  tasks  should  be  attempted  at  a  time.  Experi¬ 
ence  indicates  that  organizations  cannot  effectively  implement  more  than  a  few  major 
changes  at  one  time. 
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A  timetable  for  staffing  the  action  plan  team  should  be  completed  within  one  or  two  months 
of  the  on-site  presentation  of  the  final  assessment  report.  The  development  of  the  action 
plan  itself  should  be  completed  within  three  to  six  months  of  the  presentation.  Periodic  and 
frequent  (e.g.,  biweekly  or  monthly)  senior  management  updates  should  occur  until  the  ac¬ 
tion  plan  team  is  fully  staffed  and  functioning.  This  safeguard  ensures  that  the  proper  prior¬ 
ity  is  given  to  the  important  job  of  formulating  concrete  improvement  plans. 

Incremental  Approach  to  Large  Changes 

For  action  tasks  that  require  major  changes,  the  following  incremental  approach  to  change 
is  recommended: 

1 .  Select  a  pilot  project  for  implementation. 

2.  Implement  the  action  task  on  the  pilot  project. 

3.  Evaluate  the  result  of  the  pilot  change  effort. 

4.  Revise  the  action  task  implementation  plan  based  on  feedback  from  the  pilot 
project. 

5.  Implement  the  action  task  on  a  broader  basis. 

This  gradual  approach  to  major  change  efforts  reduces  the  risk  of  failure,  and  smaller  early 
successes  can  pave  the  way  for  larger  successes  as  the  effort  builds  momentum  via  accep¬ 
tance  and  participation  throughout  the  organization. 


8.2.  Action  Plan  Review 

After  the  action  plan  is  formulated,  it  should  be  reviewed  in  order  to  do  the  following: 

•  Improve  the  quality  of  the  action  plan. 

•  Provide  opportunities  for  participation  in  the  action  plan  formulation  by  prac¬ 
titioners  and  senior  management,  and  to  help  enroll  them  into  the  change  proc¬ 
ess. 

•  Provide  a  mechanism  to  build  consensus  to  the  action  plan. 

•  Help  to  build  and  confirm  the  commitment  of  senior  management  to  the  action 
plan,  as  well  as  the  resources  necessary  to  implement  it. 

Three  reviews  are  recommended:  no  order  is  implied  except  that  a  site  peer  review  should 
come  first: 

•  Site  peer  review 

•  Site  senior  management  review 

•  SEI  review 


32 


CMU/SEI-89-TR-7 


Site  Peer  Review 

Key  management  and  technical  professionals  from  the  organizational  units  that  will  imple¬ 
ment  the  action  plan  should  participate  in  a  peer  review  of  the  plan.  This  review  serves  two 
purposes:  1)  It  gives  these  professionals  the  opportunity  to  provide  their  expertise,  insight, 
and  suggestions  for  improving  the  action  plan  and  its  subsequent  implementation;  and  2)  By 
involving  more  professionals  from  the  organization,  the  review  helps  facilitate  the  accep¬ 
tance  of  the  planned  software  process  improvement  effort. 

This  review  culminates  in  a  set  of  issues  and  suggestions  for  improving  the  plan.  The  action 
plan  team  then  revises  the  plan,  based  on  the  guidance  of  the  peer  review. 

Site  Senior  Management  Review 

Site  senior  management  reviews  the  action  plan  by  examining  it  from  at  least  these 
perspectives: 

1 .  Does  the  plan  balance  the  need  for  software  process  improvements  with  the 
economic  necessity  of  simultaneously  producing  software  products? 

2.  Does  the  plan  identify  the  appropriate  level  of  resources  (people  and  dollars) 
to  implement  the  proposed  plan?  Is  the  time  for  implementing  the  plan  res'is- 
tic? 

3.  Are  the  action  task  checkpoints  concisely  identified  and  clearly  stated? 

If  the  action  plan  addresses  these  and  other  questions  specific  to  the  assessed  organiza¬ 
tion,  senior  management  gives  approval  to  proceed  with  action  plan  implementation.  If  not, 
the  plan  is  revised  to  address  senior  management  concerns.  In  any  case,  senior  manage¬ 
ment  signifies  its  approval  to  proceed  with  implementation  by  committing  the  needed 
resources  and  personnel  to  the  software  process  improvement  effort. 

SEI  Review 

The  SEI  review  is  conducted  by  a  team  of  experienced,  SEI  software  process  professionals. 
At  least  one  SEI  assessment  team  member  will  participate  in  this  review,  as  one  who  is 
familiar  with  the  software  process  improvement  needs  of  the  assessed  organization.  The 
SEI  team  reviews  the  action  plan,  focusing  on  its  feasibility  and  practicality,  the  potential  for 
improving  the  assessed  organization's  software  process,  and  the  resources  and  schedule 
planned  for  the  improvement  effort.  This  review  results  in  a  set  of  issues  and  suggestions 
for  improving  the  action  plan.  These  are  communicated  to  the  action  plan  team  leader  in  a 
letter  from  the  SEI  review  team.  This  review  does  not  constitute  an  approval  to  proceed  with 
action  plan  implementation.  Only  the  assessed  organization’s  senior  management  can  give 
this  approval. 
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8.3.  Periodic  Site-Management  Review 

Action  plan  implementation  formally  begins  when  senior  management  approves  the  plan. 
There  is  a  risk  with  assessments  and  the  subsequent  action  plan  preparation:  after  these 
activities  are  completed,  only  superficial  and  ineffective  change  efforts  may  be  made  to  initi¬ 
ate  software  process  improvement.  Experience  in  these  cases  indicates  that  the  organi¬ 
zation  will  soon  revert  back  to  "business  as  usual,"  and  the  enthusiasm  and  momentum  gen¬ 
erated  by  the  assessment  and  action  plan  preparation  will  dissipate.  For  process  improve¬ 
ment  to  be  successful,  all  organizational  units  need  a  high  level  of  commitment,  sustained 
change  effort,  and  focus  on  improvement. 

Some  catalyst  is  required  to  maintain  focus  on  the  improvement  process.  One  highly  effec¬ 
tive  means  is  periodic  management  review.  This  is  accomplished  by  establishing  a  clear  set 
of  improvement  goals  to  be  achieved  by  the  time  of  the  next  assessment  and  defining  action 
task  checkpoints  to  be  reached  in  each  of  the  intervening  quarters. 

When  action  plan  implementation  begins,  it  is  recommended  that  periodic  management  re¬ 
views  be  held  monthly  for  the  first  quarter  or  until  implementation  gets  under  way  and  be¬ 
comes  more  stable.  Quarterly  management  reviews  are  recommended  thereafter.  These 
reviews  are  scheduled  with  senior  management,  and  the  objectives  are  defined  in  advance. 
The  software  engineering  process  group  (see  Section  8.4)  conducts  these  reviews  with  sen¬ 
ior  management.  These  items,  and  perhaps  others,  are  covered  in  a  periodic  management 
review: 

•  Resource  staffing  actuals  against  action  plan  staffing  requirements. 

•  Progress  to  date  on  scheduled  action  task  checkpoints. 

•  Resolution  of  high-priority  process  improvement  issues. 

Other  items  may  be  covered,  depending  on  the  specific  needs  of  the  assessed  organization. 


8.4.  Software  Engineering  Process  Group 

One  premise  of  process  management  is  that  the  quality  of  a  product  is  governed  by  the 
process  used  to  produce  it.  The  quality  of  the  software  process  will  not  improve  by  itself; 
resources  and  people  are  needed.  The  software  engineering  process  group  (SEPG)  is  the 
focal  point  for  software  process  improvement  within  a  software  organization.  It  provides  a 
center  of  expertise  to  assist  software  practitioners  in  the  organization  and  serves  as  the 
vehicle  for  organizational  learning  and  change  advocacy. 

An  SEPG  is  chartered  to  facilitate  the  definition,  documentation,  and  improvement  of  the 
organization’s  software  process.  Its  ongoing  functions  include: 

•  Performing  periodic  software  process  assessments. 

•  Reporting  status  of  the  software  process  to  senior  management. 

•  Facilitating  the  definition  and  improvement  of  technical  and  management  proc¬ 
esses. 
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•  Facilitating  the  definition  and  maintenance  of  process  standards. 

•  Establishing  and  maintaining  a  software  process  database. 

•  Initiating  and  providing  process  education  and  training. 

•  Identifying,  screening,  and  evaluating  appropriate  candidate  software  technol¬ 
ogies. 

•  Providing  process  consultation  to  software  practitioners. 

Once  the  SEPG  is  established,  the  group  also  actively  participates  in  the  action  plan  as¬ 
pects  of  the  process  improvement  effort  by  doing  the  following: 

•  Facilitating  action  plan  development  and  review  efforts. 

•  Leading  and  coordinating  action  plan  implementation. 

•  Establishing  and  monitoring  pilot  change  efforts. 

•  Tracking  action  plan  implementation  progress  against  plan. 

•  Conducting  periodic  management  reviews. 

Additional  areas  of  activity  may  be  appropriate.  These  will  depend  on  the  specific  findings 
and  recommendations  of  the  assessment  team.  Smooth  transition  and  coordination  be¬ 
tween  the  action  plan  team  and  a  newly  established  SEPG  are  necessary  for  effective  im¬ 
plementation  of  the  action  plan. 

The  SEPG  is  a  small  but  dedicated  resource  of  competent  and  experienced  professionals. 
Typically,  2  to  3  percent  of  the  size  of  the  organization  is  adequate.  While  some  personnel 
should  be  permanently  assigned  to  the  SEPG,  it  is  useful  to  rotate  technical  and  manage¬ 
ment  professionals  into  the  SEPG  for  two-  to  three-year  assignments. 

Note  that  the  SEPG  function  is  not  to  be  confused  with  the  SQA  function,  which  is  an  audit 
and  enforcement  mechanism.  The  SEPG  works  closely  with  and  assists  software  projects 
by  providing  knowledge,  guidance,  and  consultation  on  software  technologies,  methods, 
practices,  and  tool*. 


8.5.  SEI  Consultation  and  Guidance 

Within  resource  constraints,  the  SEI  provides  post-assessment  consultation  and  guidance  to 
an  assessed  organization.  Typically  this  involves  the  SEI  peer  review,  telephone  consulting, 
ongoing  affiliate  relationships  between  the  organization  and  the  SEI,  and  possibly  self- 
assessment  training  (see  next  section).  The  extent  and  nature  of  this  involvement  will  be 
determined  on  an  individual  basis. 
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8.6.  Self-Assessments 

A  software  process  assessment  is  not  a  one-time  effort.  Ongoing  assessments  arc  an  im¬ 
portant  element  of  a  continuous  software  process  improvement  effort  within  a  software  or¬ 
ganization.  Due  to  resource  constraints,  the  SEI  is  able  to  assist  only  a  limited  number  of 
organizations  in  conducting  the  organization's  first  software  process  assessment  via  SEI- 
assisted  assessment  as  described  in  this  document. 

Organizations  should  generally  plan  to  conduct  a  follow-up  assessment  approximately 
eighteen  months  after  the  action  plan  has  been  approved  for  the  initial  SEI-assisted  assess¬ 
ment.  This  follow-up  has  the  following  purposes: 

•  Assess  the  progress  that  has  been  made. 

•  Establish  a  target  for  completion  of  the  most  important  actions. 

•  Establish  new  priorities  for  continued  improvement. 

It  is  recommended  that  organizations  use  a  self-assessment  methodology  in  follow-up  as¬ 
sessments.  Self-assessments  are  conducted  and  structured  in  a  manner  similar  to  SEI- 
assisted  assessments.  Organizations  conduct  them  using  their  own  resources  and  software 
experts.  The  SEI  has  developed  an  in-depth  training  course  for  preparing  assessment 
teams  in  this  self-assessment  methodology.  The  SEI  offers  this  training  quarterly  in  Pitts¬ 
burgh  to  organizations  that  wish  to  use  it. 
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Appendix  A:  Assessment  Agreement 

This  Agreement  is  entered  into  by  and  between  the  Software  Engineering  Institute  (SEI)  and 

_ (hereinafter  referred  to  as  Affiliate)  with  its  principal  office 

located  at _ .  This  Agreement  will  be  effective  when  executed  by  both 

parties.  Its  duration  will  be  limited  by  the  nature  and  scope  of  the  undertakings  set  forth  in  the 
Agreement.  Certain  commitments  will  survive  completion  of  the  basic  tasks  set  forth. 

The  SEI  is  a  federally  funded  research  and  development  center  (FFRDC)  owned  and  operated  by 
Carnegie  Mellon  University  pursuant  to  contract  with  the  Department  of  Defense  (DoD).  Included  in 
the  SEI  Charter  is  a  requirement  that  the  SEI  shall  establish  standards  of  excellence  for  software 
engineering  practice.  In  furtherance  of  that  requirement,  the  SEI  is  conducting  assessments  of  soft¬ 
ware  engineering  practice  with  individual  companies  and  studying  industry-wide  practices  of  software 
engineering  (the  Assessment  Program).  Affiliate  has  a  vested  interest  in  assessing  its  level  of  soft¬ 
ware  engineering  practice  on  both  an  absolute  and  comparative  basis. 

Therefore  the  parties  hereto  agree  as  follows: 

1.  Affiliate  agrees  to  participate  with  the  SEI  in  an  assessment  of  the  Affiliate's  software 
engineering  practices. 

2.  The  specific  results  of  the  Affiliate  assessment  shall  be  proprietary  to  Affiliate  to  be 
used  by  Affiliate  as  it  chooses  (with  appropriate  credit  to  the  SEI)  subject  to  the  right  of 
the  SEI  to  use  such  results  as  hereinafter  described. 

3.  Confidentiality  is  essential  to  the  success  of  the  SEI  Assessment  Program.  The  SEI  will 
not  release  or  otherwise  identify  the  results  of  any  organization’s  assessment. 

4.  The  SEI  is  free  to  use  assessment  data  and  conclusions  to  be  derived  therefrom  for 
statistical,  analytical  or  reporting  purposes  provided  that  the  confidentiality  requirement 
can  be  honored  and  that  the  information  can  be  used  without  attribution  to  its  source 
either  directly  or  by  inference. 

5.  The  SEI  will  not  publish  collective  data  externally  unless  such  data  is  based  upon  infor¬ 
mation  from  not  less  than  10  different  organizations. 

6.  The  senior  manager  of  the  segment  of  Affiliate  to  be  assessed  will  actively  participate  in 
the  assessment  by  agreeing  to  be  present  (on  mutually  acceptable  dates)  for  the  open¬ 
ing  on-site  assessment  briefing  and  the  on-site  review  of  final  findings  and  recommen¬ 
dations.  This  level  of  participation  is  critical  to  the  success  of  the  project. 

7.  The  SEI  will  provide  four  to  six  professionals  as  assessment  team  members. 

8.  The  Affiliate  will  provide  one  or  two  professionals  as  assessment  team  members.  It  is 
expected  that  one  of  these  professionals  will,  in  addition,  be  assigned  the  following 
responsibilities: 

a.  participate  in  a  one  or  two  day  working  session  at  the  SEI  shortly  following  the 
on-site  assessment  period  to  prepare  the  final  assessment  report. 

b.  develop  the  organization’s  action  plan  based  upon  the  final  assessment  report. 

9.  There  will  be  one  day  of  pre-assessment  team  training  for  the  affiliates  assessment 
team  members. 

1 0.  There  will  be  two  days  of  assessment  team  training  at  the  SEI  for  the  entire  assessment 
team. 

1 1 .  The  assessment  by  the  SEI/ Affiliate  team  at  the  Affiliate  site  will  be  done  in  three  or  four 
days. 
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12.  Typically,  three  to  five  projects  will  be  reviewed  during  the  on-site  assessment  period. 
The  Affiliate  will  ensure  that  none  of  the  projects  selected  for  review  is  involved  in  a 
competitive  phase  of  a  procurement  during  the  on-site  assessment  period. 

13.  The  SEI  as  soon  as  practicable  thereafter  will  provide  to  Affiliate  a  final  assessment 
report  which  will  include  recommended  actions.  The  SEI  does  not  represent  any  U.S. 
Government  procurement  agency;  therefore,  the  recommendations  made  by  the  SEI 
are  not  to  be  construed  as  contractual  directions  or  constructive  changes  to  contracts 
which  may  exist  between  the  Affiliate  and  the  U.S.  Government. 

14.  The  Affiliate  will  within  60  days  thereafter  provide  a  review  for  the  SEI  of  its  action  plan 
based  on  the  final  assessment  report.  The  Affiliate  is  the  sole  determinant  as  to  the 
extent  that  recommended  actions  are  implemented  through  its  action  plan.  It  is  ex¬ 
pected  that  the  review  will  cover  the  reasons  for  rejection  of  any  final  assessment  report 
recommended  actions  not  incorporated  into  the  action  plan. 

15.  The  SEI,  at  its  option,  and  with  the  consent  of  the  Affiliate,  may  continue  periodic  inter¬ 
action  with  the  Affiliate  during  implementation  of  the  action  plan. 

16.  Each  participating  organization  will  be  responsible  for  its  personnel  and  their  expenses. 
Each  will  provide  to  the  other  reasonable  working  space  and  support  services  for  as¬ 
sessment  team  activities. 

17.  If  Affiliate  discloses  identified  proprietary  information  to  the  SEI,  such  information  should 
be  disclosed  pursuant  to  a  separate  nondisclosure  agreement.  The  execution  of  such 
agreement  notwithstanding,  the  parties  agree  to  respect  the  confidential  nature  of  this 
Agreement  and  all  exchanges  of  information  hereunder.  All  confidential  and  proprietary 
information  shall  be  treated  with  at  least  the  degree  of  care  with  which  each  party  treats 
its  own  such  information. 

18.  When  Affiliate  has  received  an  assessment  report  from  the  SEI  and  has  in  turn  re¬ 
viewed  its  action  plan  with  the  SEI,  the  basic  assessment  tasks  will  be  complete  and 
this  Agreement  will  have  served  its  primary  purpose.  However,  any  commitments 
which  by  their  nature  are  intended  to  survive  termination  of  the  Agreement  (such  as 
nondisclosure)  will  survive. 

19.  This  agreement  shall  not  constitute,  create,  give  effect  to,  or  otherwise  imply  a  joint 
venture,  partnership  or  formal  business  organization  of  any  kind.  Each  party  to  this 
Agreement  shall  act  as  an  independent  contractor  and  not  as  an  agent  for  the  other, 
and  neither  party  shall  have  any  authority  to  bind  the  other  except  to  the  extent,  if  any, 
specifically  provided  herein  or  by  other  written  mutual  agreement  of  the  parties. 

20.  The  SEI  and  Affiliate  agree  that  they  are  each  fully  responsible  for  their  and  their 
employees'  actions  under  this  agreement  and  that  they  each  indemnify  and  hold  the 
other  harmless  for  any  such  actions. 


Software  Engineering  Institute  Affiliate 

By _  By _ 

Date _  Date _ 

I  have  read  and  understood,  and  I  agree  to  abide  by  the  provisions  of  this  agreement. 

Assessment  Team  Member  Signature  _ 

Date  _  _ 
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Appendix  B:  Example  Assessment  Team  Training 
Schedule 


Day  1 


Start  Time 

Activity  or  Discussion  Topic 

8:45  am 

Welcome  &  Introductions 

8:50  am 

Question  &  Answer  Period 

9:00  am 

Team-Building  Exercise 

9:30  am 

Process  Management  Overview 

10:30  am 

Break 

1 0:45  am 

Organizational  Briefing 

1 1 :30  am 

Assessment  Process  Overview 

12:15  pm 

Lunch 

1:15  pm 

Use  of  the  Maturity  Questionnaire 

2:30  pm 

Review  of  Project  Responses 

3:30  pm 

Break 

3:45  pm 

Assessment  Discussions 

4:30  pm 

Critique/Summary 

5:00  pm 

Adjourn 

Day  2 


8:30  am 

Team-Building  Exercise 

10:30  am 

Break 

10:45  am 

Assessment  Findings 

1 1 :15  am 

Final  Report  Preparation 

1 1 :45  am 

Lunch 

12:45  pm 

Planning  for  On-Site  Period  (Projects) 

2:45  pm 

Break 

3:00  pm 

Question  &  Answer  Period 

3:45  pm 

Planning  for  On-Site  Period  (Tasks) 

4:30  pm 

Training  Session  Critique 

5:00  pm 

Adjourn 

• 

Appendix  C 

:  Example  On-Site  Assessment  Schedule 

Day  1 

Start  Time 

Activity  or  Discussion  Topic 

8:00  am 

Assessment  Overview 

9:00  am 

Assessment  Briefing 

9:30  am 

Break 

9:45  am 

Questionnaire  Data  Analysis 

12:00  pm 

Lunch 

1 :00  pm 

Project  Discussions 

6:00  pm 

First  Day  Wrap-up 

Day  2 

8:00  am 

Quality  Assurance  &  Release  j 

9:45  am 

Break 

10:00  am 

Software  Integration  &  Test 

1 1 :45  am 

Lunch 

1 :00  pm 

Code  &  Unit  Test 

2:45  pm 

Break 

3:00  pm 

Requirements  &  Design 

4:45  pm 

Break 

5:00  pm 

Preliminary  Findings  &  Wrap-up 

Day  3 

8:00  am 

Project  Discussions 

12:00  pm 

Lunch 

1 :00  pm 

Findings  Formulation 

7:00  pm 

Third  Day  Wrap-up 

Day  4 

7:30  am 

Assessment  Team  Session 

9:00  am 

Project  Composite  Feedback 

1 0:30  am 

Prepare  Final  Findings  Presentation 

12:00  pm 

Lunch 

1 :00  pm 

Final  Assessment  Findings  Presentation 

3:00  pm 

Senior  Management  Meeting 

A 

3:45  pm 

Recommendations  Formulation 

w 

5:00  pm 

Adjourn  -  Assessment  Completed 
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Appendix  D:  Assessment  Questions  and  Answers 

What  is  a  software  process  assessment? 

An  appraisal  or  review  done  by  a  trained  team  of  software  professionals  to  determine  the 
state  of  an  organization’s  current  software  process,  to  determine  the  high-priority  software 
process  issues  that  face  an  organization,  and  to  start  the  actions  needed  for  software  proc¬ 
ess  improvement. 

What  is  the  difference  between  a  software  process  assessment  and  a  software  capa¬ 
bility  evaluation? 

The  methods  share  the  goal  of  facilitating  lasting  improvements  in  organizations’  software 
process;  but  given  how  the  results  of  the  methods  are  used,  each  has  different  objectives. 
The  main  objective  of  a  capability  evaluation  is  to  identify  software  engineering  and  man¬ 
agement  risks  of  contractors  who  are  either  bidding  on  a  proposal  or  performing  on  contract. 
As  in  assessments,  the  method  identifies  strengths  and  weaknesses  of  an  organization's 
current  software  process.  In  addition,  capability  evaluation  considers  the  extent  of  an 
organization’s  software  process  improvement  efforts.  On  the  other  hand,  the  main  objective 
of  an  assessment  is  to  determine  the  key  software  process  issues  facing  an  organization 
and  start  the  changes  needed  for  improvement. 

What  is  the  difference  between  an  assessment  and  an  audit? 

Again,  the  major  difference  is  in  objectives.  The  overall  objective  of  an  audit  is  to  independ¬ 
ently  review  for  compliance  and  adherence  to  some  established  set  of  criteria.  The  major 
objective  of  an  assessment  is  to  determine  the  key  software  process  issues  facing  an  organ¬ 
ization  and  start  the  changes  needed  for  improvement. 

How  many  SEI-assisted  assessments  has  the  SEI  conducted? 

As  of  the  date  of  this  document,  the  SEI  has  conducted  ten  SEI-assisted  assessments. 

How  many  SEI-assisted  assessments  does  the  SEI  do  every  year? 

The  SEI  performs  approximately  six  SEI-assisted  assessments  per  year. 

Do  organizations  pay  the  SEI  for  conducting  SEI-assisted  assessments? 

No,  not  currently. 

How  do  I  go  about  having  an  SEI-assisted  assessment  of  my  organization? 

Contact  the  SEI  and  request  an  SEI-assisted  assessment.  Chapter  3  of  this  report  provides 
information  on  the  selection  criteria  for  SEI-assisted  assessments.  If  an  SEI-assisted  as¬ 
sessment  isn't  possible,  self-assessment  training  may  be  arranged. 
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Why  do  organizations  want  SEI-assisted  assessments? 

Most  competitive  organizations  are  interested  in  improving  themselves.  Assessments  are  a 
way  to  characterize  the  state  Of  the  software  process  in  an  organization  and  identify  the 
most  important  software  issues  facing  an  organization,  thus  helping  an  organization  focus 
on  software  process  improvement. 

Is  the  SEI  restricted  to  DoD  contractors  when  selecting  candidates  for  SEI-assisted 
assessments? 

No,  the  SEI  is  not  officially  restricted  to  selecting  only  DoD  contractors.  However,  in  keeping 
with  its  mission,  the  SEI  gives  primary  consideration  to  the  needs  of  the  mission-critical 
computer  resource  (MCCR)  community. 

Are  the  results  of  assessments  used  for  evaluations  by  the  DoD? 

No,  the  data  is  kept  confidential  in  accordance  with  the  assessment  agreement.  If  an  organ¬ 
ization  has  had  an  assessment,  it  will  still  have  to  do  the  evaluation  separately.  A  contractor 
evaluation  is  a  separate  activity  from  an  assessment. 

Does  the  SEI  have  any  concrete  data  that  supports  or  proves  that  conducting  assess¬ 
ments  actually  helps  software  process  Improvement? 

Current  evidence  seems  to  indicate  that  assessments  fit  into  software  process  improvement 
very  well.  Assessments  are  based  on  the  idea  that  an  organization  must  know  where  its 
software  process  is  before  it  can  start  to  improve  the  process.  However,  there  currently  is 
no  data  that  proves  this. 

Can  an  organization  achieve  software  process  Improvement  without  using  the  soft¬ 
ware  process  maturity  model  and  the  software  process  assessment  method? 

It  may  be  possible,  but  having  a  documented  strategy  for  improvement  allows  an  organi¬ 
zation  to  benefit  from  lessons  learned  and  improve  the  strategy  over  time.  The  software 
process  maturity  model  and  the  software  process  assessment  method  comprise  one  docu¬ 
mented  approach  to  improvement. 

Who  at  the  organization  should  select  the  projects  and  the  functional  area 
representatives? 

It  doesn’t  really  matter  who  does  the  selecting  as  long  as  the  selection  criteria  given  in  this 
document  is  followed  and  there  is  agreement  among  the  organization’s  management  chain 
(the  project  manager  up  to  the  senior  site  representative). 
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What  does  the  SEI  mean  by  confidentiality?  Who  at  the  SEI  sees  the  final  reports  of 
an  SEl-assisted  assessment? 

Every  organization  defines  confidentiality  differently.  In  the  case  of  the  SEI,  the  assessment 
final  reports  are  treated  as  special  reports.  All  staff  members  must  abide  by  the  confiden¬ 
tiality  provisions  in  the  assessment  agreement.  A  special  report  is  never  available  to  any¬ 
one  except  for  those  listed  below: 

1.  The  SEI  assessment  team  members. 

2.  Process  Program  members,  including  secretaries  who  are  required  to  assist 
with  the  actual  creation  of  the  report. 

3.  SEI  Information  Management,  usually  one  professional  editor  and  a  document 
production  specialist. 

4.  The  SEI  director. 

5.  Persons  at  the  SEI  who  have  a  "need  to  know"  and  have  the  approval  of  Proc¬ 
ess  Program  director. 

6.  Process  Program  staff  who  are  doing  data  analysis. 

What  are  the  phases  of  the  SEl-assisted  assessment  and  how  long  are  they? 

The  six  SEl-assisted  assessment  phases  are: 

1.  Selection  Phase  -  This  time  is  determined  by  the  SEI  and  it  does  vary.  The 
selection  criteria  in  Chapter  3  must  be  satisfied.  Atter  selection,  the  organi¬ 
zation  may  be  placed  on  a  waiting  list. 

2.  Commitment  Phase  -  The  organization  determines  the  length  of  this  phase 

3.  Preparation  Phase  -  This  phase  takes  about  two  months  and  includes  the  time 
needed  for  the  selection  of  the  assessment  team  and  the  assessment  partic¬ 
ipants,  the  half-day  pre-assessment  team  training  presentation,  the  two-day 
assessment  team  training,  and  the  preparation  for  the  assessment. 

4.  Assessment  Phase  -  The  actual  assessment  takes  four  days. 

5.  Report  Phase  -  This  phase  takes  about  five  to  six  weeks.  It  includes  the  time 
needed  to  write  the  report,  the  time  for  two  report  reviews,  and  the  time  for 
professional  editing.  Also  included  is  the  time  for  the  recommendations  pres¬ 
entation. 

6.  Assessment  Follow-up  Phase  -  The  time  needed  for  action  plan  development 
depends  on  the  organization  but  usually  is  three  to  six  months.  Executing  the 
action  plan  takes  the  greatest  amount  of  time.  Currently  there  is  little  data  on 
execution,  but  an  estimate  is  about  eighteen  months  before  another  assess¬ 
ment  is  needed. 

Phases  2-5,  which  are  the  heart  of  the  assessment,  take  about  five  months.  To  summarize, 
assessments  are  a  lot  of  work,  and  a  great  deal  of  preparation  and  follow-up  are  necessary 
for  an  assessment  to  be  successful. 


CMU/SEI-89-TR-7 


47 


What  do  the  results  of  an  assessment  really  look  like? 

The  output  of  an  assessment  is  basically  the  final  report.  The  report  documents  the  assess¬ 
ment  findings  and  recommendations,  along  with  the  details  of  the  assessment.  Since  as¬ 
sessments  are  preludes  to  action,  other  results  include  the  initiation  of  the  changes  that  are 
necessary  for  software  process  improvement.  Action  plan  formulation  and  execution  are 
also  results  of  an  assessment. 
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Appendix  E:  Assessment  Glossary  of  Terms 

action  plan:  A  plan  that  addresses  how  to  implement  the  recommendations  in  an  assess¬ 
ment  final  report,  including  the  detailed  actions  and  the  change  necessary.  See  action  task. 

action  task:  The  part  of  an  action  plan  that  addresses  how  a  specific  recommendation  or 
recommendations  in  the  assessment  final  report  will  be  implemented. 

assessment  agreement:  An  agreement  between  the  SEI  and  the  organization  to  be  as¬ 
sessed  that  states  the  terms  and  conditions  of  the  SEI-assisted  assessment. 

assessment  findings:  The  assessment  team's  view  of  the  most  important  software  proc¬ 
ess  issues  or  problems  facing  an  organization. 

assessment  follow-up  phase:  The  last  assessment  phase,  which  consists  of  formulating 
the  action  plan  from  the  assessment  final  report,  executing  the  action  plan  to  implement  the 
assessment  recommendations,  and  striving  for  software  process  improvement. 

assessment  process:  The  assessment  process  is  generally  made  up  of  five  phases:  com¬ 
mitment,  preparation,  on-site  assessment,  report,  and  assessment  follow-up.  The  SEI- 
assisted  assessment  process  adds  a  sixth  phase  to  the  beginning  of  the  assessment  proc¬ 
ess:  the  selection  phase. 

assessment  recommendations:  The  description  of  the  actions  to  be  taken  to  correct  or 
improve  the  situations  identified  in  the  assessment  findings  and  their  consequences. 

assessment  recording  form:  This  form  is  used  to  record  each  yes  or  no  answer  from  the 
maturity  questionnaire  so  the  data  can  be  scored  and  analyzed. 

assessment  team:  A  team  of  experienced  software  professionals  that  are  trained  in  the 
software  process  assessment  method  to  perform  assessment(s). 

assessment  team  training:  A  two-day  training  program  for  the  assessment  team  members 
to  build  a  coherent  team  and  to  learn  the  assessment  process,  the  software  process  im¬ 
provement  cycle,  and  the  software  process  assessment  methodology. 

change  agent:  An  individual  or  group  that  has  sponsorship  and  is  responsible  for  im¬ 
plementing  or  facilitating  change(s).  An  example  of  a  change  agent  is  the  software  engi¬ 
neering  process  group.  This  is  in  contrast  to  a  change  advocate,  who  is  an  individual  or 
group  who  wants  to  achieve  a  change  but  lacks  sufficient  sponsorship. 

change  sponsor:  An  individual  or  group  who  legitimizes,  authorizes,  and  supports  the 
change.  In  an  assessment,  this  person  is  usually  the  senior  site  manager. 
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commitment  phase:  In  this  phase  of  the  assessment  process,  an  organization  makes  a 
commitment  to  conduct  an  assessment. 

final  report:  The  report  contains  the  findings  and  recommendations,  along  with  the  details 
of  the  assessment. 

findings:  See  assessment  findings. 

functional  areas:  Technical  areas  or  unit  operations  of  the  software  process.  For  a  typical 
assessment  these  are:  software  quality  assurance  and  release,  software  integration  and 
test,  coding  and  unit  test,  detailed  design,  requirements  and  high-level  design. 

functional  area  representatives:  The  software  professionals  representing  one  or  more  of 
the  functional  areas  for  a  software  process  assessment. 

on-site  assessment  phase:  The  phase  of  the  assessment  process  in  which  the  actual 
assessment  is  conducted  at  the  organization's  site. 

pre-assessment  team  training  presentation:  A  presentation  given  by  the  SEI  assess¬ 
ment  team  leader  or  SEI  assessment  coordinator  to  the  site  assessment  team  members  to 
prepare  them  for  the  assessment  team  training. 

preparation  phase:  This  phase  of  the  assessment  process  is  devoted  to  preparing  for  the 
on-site  assessment. 

process:  A  set  of  actions,  tasks,  and  procedures  that  when  performed  or  executed  obtain  a 
specific  goal  or  objective. 

project  representatives:  The  software  professionals  representing  a  project  to  be  as¬ 
sessed.  These  usually  are  the  project  leader  and  any  optional  support  from  one  or  more  of 
the  project  technical  professionals. 

recommendations:  See  assessment  recommendations. 

report  phase:  The  phase  of  the  assessment  process  that  is  concerned  with  formulation 
and  communication  of  final  assessment  findings  and  recommendations.  This  is  usually  ac¬ 
complished  through  a  written  final  report  and  oral  presentation. 

scoring  templates:  The  templates  used  to  score  the  initial  software  process  maturity  level 
from  the  data  on  the  assessment  recording  forms. 

SEI-assisted  assessments:  Software  process  assessments  that  are  assisted  by  the  SEI, 
that  is,  which  are  conducted  by  an  assessment  team  with  members  from  both  the  SEI  and 
the  organization  to  be  assessed. 
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selection  phase:  The  first  phase  of  the  SEI-assisted  assessment  process,  in  which  the  SEI 
selects  a  candidate  organization  for  a  software  process  assessment. 

senior  site  manager:  The  senior  manager  who  sets  the  operational  priorities  for  the  organ¬ 
ization. 

software:  Computer  programs,  procedures,  rules,  and  possibly  associated  documentation 
and  data  pertaining  to  the  operation  of  a  computer  system.  IEEE  Standard  Glossary  of  Soft¬ 
ware  Engineering  Terminology  (SET),  1983 

software  capability  evaluation:  An  evaluation  or  review  of  a  contractor’s  software  engi¬ 
neering  capability,  conducted  as  part  of  a  software-intensive  acquisition  or  risk  abatement 
program,  and  performed  by  a  trained  evaluation  team  from  the  acquisition  agency. 

software  development  process:  The  process  by  which  a  user’s  needs  are  translated  into 
software  requirements,  software  requirements  are  translated  into  design,  the  design  is  im¬ 
plemented  into  code,  and  the  code  is  tested,  documented,  and  certified  for  operational  use. 
IEEE  SET  Glossary,  1983 

software  engineering:  The  systematic  approach  to  the  development,  operation,  mainte¬ 
nance,  and  retirement  of  software.  IEEE  SET  Glossary ,  1983 

software  engineering  process:  The  total  set  of  software  engineering  activities  needed  to 
transform  a  user’s  requirements  into  software.  The  process  of  applying  engineering  prin¬ 
ciples  to  produce  software. 

software  engineering  process  group  (SEPG):  A  group  of  specialists  concerned  with  the 
software  engineering  process  used  by  an  organization.  The  SEPG  defines  and  documents 
the  software  process,  establishes  and  defines  process  metrics,  supports  project  data  gather¬ 
ing,  assists  projects  in  analyzing  data,  and  advises  management  on  areas  requiring  further 
attention. 

i  software  process:  A  process  performed  to  produce,  support,  maintain,  and  enhance  soft¬ 

ware.  Examples  of  a  software  process  are  a  software  development  process,  a  software 
maintenance  process,  etc. 

software  process  assessment:  An  appraisal  or  review  done  by  a  trained  team  of  software 
►  professionals  to  determine  the  state  of  an  organization's  current  software  process,  to  deter¬ 

mine  the  high-priority  software  process  issues  that  face  an  organization,  and  to  start  the 
actions  needed  for  software  process  improvement. 

software  process  improvement:  The  changes  implemented  to  a  software  process  that 
1  bring  about  improvements. 
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sponsor:  See  change  sponsor. 

state  of  the  practice:  The  commonly  accepted  level  of  use  of  applied  scientific  or  engi¬ 
neering  knowledge  at  a  particular  time. 

statistical  process  control:  The  use  of  statistical  techniques  to  control  the  quality  of  a 
process. 

statistical  quality  control:  The  use  of  statistical  techniques  to  control  the  quality  ot  a  prod¬ 
uct  and  process. 

workshop  assessments:  Informal  assessments  conducted  at  various  workshops,  con¬ 
ferences,  tutorials,  or  symposiums  by  the  SEI  to  gather  industry  profile  data. 
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